Before last weekend, my Debian 8 setup was relatively stable; there were some quirks (like not being able to use accelerated graphics at all), but it was usable for most of what I do. I was using the fglrx driver from AMD for my setup which is the following:

• ATI Radeon HD 5870


• 3 identical Samsung SyncMaster SA850 monitors


• 3 HBR2 DP cables - no dongles or adapters

This worked well, using the full resolution and proper refresh rate for the displays, but I wanted to update to Debian 9, get on the open source driver, and hopefully fix the accelerated graphics issues. I had no idea what I was in for. It was been a long weekend, especially since I am no guru at X. From my first time using it with slackware 0.97 until now, it has always seemed more like magic than anything else :)

Here is where I am now. The xserver-xorg-video-radeon driver appears to be working correctly, and identifies the card and the monitors. When I have the system booting into graphical mode, the system actually thinks that everything is fine (gdm3 is running, my applications start up, etc.), but there is nothing on the screens. The monitors complain that "Not Optimal Mode. Recommended Mode 2560x1440" which is the native resolution for the monitors, and what I had been using.

I had initially searched and saw that I could at least get the virtual consoles up and running if I added nomodeset to grub, or set modeset=0 in the kernel module options. However, that basically disables the video driver, and it gives up. Removing that, and leaving modeset enabled, I can use xrandr to change to 1280x1024 and that "works", but it is not the desired setup at all (and doesn't fix the issue with the virtual terminals). I have tried adding a modeline with cvt and xrandr with the exact specifications of the monitor, but that didn't work either. I even tried with 2560x1440 @ 30Hz, and nothing. I tried toggling DPMS. I even tried a "live" Ubuntu USB boot. Nothing.

All packages on the box are updated as of 2018-03-19T14:00Z.

As seems to be the normal practice for this type of plea for help, here are some package versions, configuration files, and logs.

• uname: Linux ziggy 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64 GNU/Linux
  


• debian: 9.4


• xserver-xorg: 1:7.7+19


• xserver-xorg-core: 2:1.19.2-1+deb9u2


• xserver-xorg-video-radeon: 1:7.8.0-1+b1


• firmware-misc-nonfree: 20161130-3

xorg.conf:

Section "Monitor"

  Identifier  "0-DFP1"

  Option      "TargetRefresh" "60"

  Option      "Position" "0 0"

  Option      "DPMS" "false"

  Option      "Rotate" "normal"

  Option      "Disable" "false"

EndSection



Section "Monitor"

  Identifier  "0-DFP2"

  Option      "TargetRefresh" "60"

  Option      "Position" "2560 0"

  Option      "DPMS" "false"

  Option      "Rotate" "normal"

  Option      "Disable" "false"

EndSection



Section "Monitor"

  Identifier   "0-DFP3"

  Option      "TargetRefresh" "60"

  Option      "Position" "5120 0"

  Option      "DPMS" "false"

  Option      "Rotate" "normal"

  Option      "Disable" "false"

EndSection



Section "Device"

  Identifier "Card0"

  Driver     "radeon"

  BusID      "PCI:6:0:0"

EndSection



Section "Screen"

  Identifier   "Screen0"

  Device       "Card0"

  DefaultDepth 24

  SubSection "Display"

    Viewport 0 0

    Depth    24

    Virtual  7680 1440

  EndSubSection

EndSection

The Xorg.0.log is large (available here) but here are some highlights:

[    10.141] (II) RADEON(0): Output DisplayPort-0 connected

[    10.141] (II) RADEON(0): Output DisplayPort-1 connected

[    10.141] (II) RADEON(0): Output DisplayPort-2 connected

[    10.141] (II) RADEON(0): Output DisplayPort-3 disconnected

[    10.141] (II) RADEON(0): Output DisplayPort-4 disconnected

[    10.141] (II) RADEON(0): Output DisplayPort-5 disconnected

[    10.141] (II) RADEON(0): Using spanning desktop for initial modes

[    10.141] (II) RADEON(0): Output DisplayPort-0 using initial mode 2560x1440 +0+0

[    10.141] (II) RADEON(0): Output DisplayPort-1 using initial mode 2560x1440 +2560+0

[    10.141] (II) RADEON(0): Output DisplayPort-2 using initial mode 2560x1440 +5120+0

Which looks VERY promising, but then this is what seems to kill it:

[    10.253] (II) RADEON(0): Setting screen physical size to 2032 x 381

What!?

NOTE: there are NO "EE" lines in there at all.

I have looked at many other threads, have tried using the different 3 ports on the card, using only one monitor, etc.

fglrx is no longer supported in Debian 9. Your post appears related to this question previously asked in that they are reporting issues with missing fglrx. I think this forum post has a solution that could work for you. I am including links to the official Debian wiki on how to install open source and proprietary drivers. I recommend you only use the open source drivers.

Use xrandr to Correct Configuration

Delete your old x.org configuration (rm /etc/X11/xorg.conf) and set the output with xrandr.

xrandr -q

If this does not help, I recommend you start from scratch.

Start from Scratch

To make the troubleshooting process smoother let us start by purging all old configurations and drivers. As root(sudo) run these commands:

apt purge "fglrx.*"

rm /etc/X11/xorg.conf

apt install --reinstall xserver-xorg-core libgl1-mesa-glx:i386 libgl1-mesa-dri:i386 libgl1-mesa-glx:amd64 libgl1-mesa-dri:amd64

Make sure your sources.list contains the contrib and non-free repositories like so:

deb http://ftp.us.debian.org/debian/ stable main contrib non-free 

If you needed to add them, do not forget to update apt, simply run as root(sudo) apt update. Next if you choose to use only the open source firmware for your card you will need to run this command:

apt-get install firmware-linux-nonfree libgl1-mesa-dri xserver-xorg-video-ati

And reboot your computer to initialize your graphics drivers and settings.

Conclusion

Please comment with any issues and I can update this answer to try and best correct your issue. Do not forget to reference the wiki on how to remove and install drivers. I believe this post on multi-monitor setups for Debian could help as well. Best of Luck!