Hello Everyone,
I've done some searching around on my issue, but it seems the majority of the results returned all have to do with a transcode directory/volume/path being out of space or non-writeable - and that's simply not my issue.
The issue is the dreaded (and to me, needlessly vague) error message "There was a problem playing this item". This problem seems to happen only during instances of multiple transcodes.
My plex server is rather large (120TB, primarily zfs storage on Ubuntu). The hardware itself is powerful enough to transcode multiple streams concurrently (i've had up to 15+ in the past, though things got quite sketchy with that many going beyond that). The hardware is an i7 5930k (hex-core), 32GB RAM, a couple of M1015 HBA's, with a total of 20 6TB WD Red drives connected between the HBA's and onboard SATA ports (the majority on HBAs). ZFS Dedup is disabled, and each 4 drive vdev is configured to be raidz, so 1 drive loss per vdev can be tolerated. The vdevs on the HBA's are all configured to a single pool (4 vdevs in the sas pool, 1 vdev in the 'onboard sata' pool). Write performance is fantastic on the sas pool, thanks to the multiple vdev participation, and tolerable on the sata pool. The bulk of my media lives on the sas pool. The OS is on a 512GB SSD, which is also where the plex home lives.
The machine is dedicated to plex, and I share my library with friends and family. The problems I've been having have only cropped up recently, say within the last 3 or 4 months perhaps.
I have my transcode path configured to be /var/lib/plexmediaserver/transcode (which exists on ssd, and is owned by plex:plex and has 755 perms). The plex services themselves all run as the user 'plex' and the install is a very typical one done via dpkg -i.
This latest installation is a "clean" installation, as in my zeal to fix this problem, I decided to start fresh and rebuild my libraries. Sadly this had no effect.
It seems that no matter where I configure my transcode temp dir to be, I always have people running into the "there was a problem playing this item" error - despite there being plenty of space and permissions being plex:plex 755 for all directories involved.
The first and most obvious thing was to check perms and space on the tmp dir. I've tried putting the transcode tmp on both on the sas pool as well as SSD. The result is the same. Plenty of storage available, permissions good, but after 8 or so concurrent transcodes, people randomly get the "there was a problem playing this item". During that time CPU usage is normally elevated (maybe 50% across cores) but certainly not pegged. I've seen this issue happen with CPU across all cores at around 15% utilization, so I strongly doubt it's a CPU constraint.
I've found that having SRT's downloaded for my media items also randomly causes this (or causes it to happen more frequently, probably because it means the difference between leveraging directplay versus transcoding). In one instance, deleting the SRT file (and waiting a few moments for plex to pick up the now-deleted srt file) "fixed" the issue (even when forced transcode is enabled) - though I've since removed all SRT files in the interest of troubleshooting. Regardless, certain clients (usually roku devices) still randomly get this "there was a problem playing this item" message.
It's important to note that if people "keep trying", eventually the video will play - so "brute forcing" the problem eventually gets people's content to work - but 19 times out of 20, it fails.
It is almost as if something is artificially constraining the amount of concurrent transcodes within Plex itself.
Has anyone with a large plex library with multiple concurrent transcodes being the norm ever experienced this? It is extremely frustrating. I can provide whatever troubleshooting logs are required, but thus far I haven't seen anything that stands out.
Thanks in advance,
-K