I'm configuring & compiling new 3.0 kernel. One of the goodies I planned to use for some time (by patching) that was merged into 3.0 is zram.

Is it possible to set both hdd swap and zram swap so the zram is used first and only spilled pages are put into actual swap?

Sidenote: because of per-cpu locking, it is important to have as many zram-swaps as CPUs (modprobe zram_num_devices=n zram) instead of a single big one. RTFM!

swapon have -p switch which sets the priority. I can set up:

swapon -p 32767 /dev/zram0

swapon -p 0 /dev/my-lvm-volume/swap

Or in /etc/fstab:

/dev/zram0              none swap sw,pri=32767 0 0

/dev/my-lvm-volume/swap none swap sw,pri=0     0 0

EDIT: Just for a full solution - such line may be helpful as udev rule:

KERNEL=="zram0", ACTION=="add", ATTR{disksize}="1073741824", RUN="/sbin/mkswap /$root/$name"

Select how many zram devices do you want by passing

zram.num_devices=4

to kernel parameters or directly to module (without zram.) . Default value is 1. zram will be able utilize as many cpus as number of devices you create.

Prepare zram0 device, set size to 1GB.

echo 1000000000 > /sys/block/zram0/disksize

Alternatively you can do it by udev rule.
Create swap on zram0

mkswap /dev/zram0

and enable it with higher priority than other swap devices

swapon -p 32767 /dev/zram0

When you enable zram, it will automatically have priority.

For ubuntu 16.04: /usr/bin/init-zram-swapping

You can edit that file (make a backup 1st), to lower the used real ram, I changed the mem line to this:

mem=$(((totalmem / 4 / ${NRDEVICES}) * 1024))

For some reason there seems to be a lot of misinterpretation of https://www.kernel.org/doc/Documentation/blockdev/zram.txt

It clearly states
"2) Set max number of compression streams
Regardless the value passed to this attribute, ZRAM will always
allocate multiple compression streams - one per online CPUs - thus
allowing several concurrent compression operations. The number of
allocated compression streams goes down when some of the CPUs
become offline. There is no single-compression-stream mode anymore,
unless you are running a UP system or has only 1 CPU online.

To find out how many streams are currently available:
cat /sys/block/zram0/max_comp_streams"

But repetitively there is a common urban myth max streams is 1.

Its plainly not true.

The two OS where zram has proven effective Chrome OS & Android you a single device.
Also they tweak page-cluster

page-cluster controls the number of pages up to which consecutive pages
are read in from swap in a single attempt. This is the swap counterpart
to page cache readahead.
The mentioned consecutivity is not in terms of virtual/physical addresses,
but consecutive on swap space - that means they were swapped out together.

It is a logarithmic value - setting it to zero means "1 page", setting
it to 1 means "2 pages", setting it to 2 means "4 pages", etc.
Zero disables swap readahead completely.

The default value is three (eight pages at a time). There may be some
small benefits in tuning this to a different value if your workload is
swap-intensive.

Lower values mean lower latencies for initial faults, but at the same time
extra faults and I/O delays for following faults if they would have been part of
that consecutive pages readahead would have brought in.

echo "0" > /proc/sys/vm/page-cluster to force single page

Much seems to originate from zram_config the debian/ubuntu package that for some reason seems to have very little correlation with the kernel documents for zram and has bred a series of Chinese whispers that in essence could be completely wrong.

With file swap do you create a swap drive for each core? Maybe that might answer your questions.
Also to back this up Googles Chrome OS & Android which successfully with the above page-cluster as its not matching a disk so latency can be improved.

Also "num_devices parameter is optional and tells zram how many devices should be pre-created. Default: 1"

All you need to do is check if the zram sys class exists

ZRAM_SYS_DIR='/sys/class/zram-control'
if [ ! -d "${ZRAM_SYS_DIR}" ]; then
modprobe zram
# create /dev/zram0
# as /sys/class/zram-control/hot_add = 1
fi

Next zram device = /dev/zram${hot_add}

Add/remove zram devices

zram provides a control interface, which enables dynamic (on-demand) device
addition and removal.

In order to add a new /dev/zramX device, perform read operation on hot_add
attribute. This will return either new device's device id (meaning that you
can use /dev/zram) or error code.

Example:
cat /sys/class/zram-control/hot_add
1

To remove the existing /dev/zramX device (where X is a device id)
execute
echo X > /sys/class/zram-control/hot_remove