It's been noted in a couple of other threads that libreoffice applications do not always group together as defined in a users ~/.fluxbox/apps file.
With xprop, the class name for libreoffice will be along the lines of:
WM_CLASS(STRING) = "libreoffice", "libreoffice-calc"
which is normally what would be used in the ~/.fluxbox/apps file.
However, while tracking down another issue with libreoffice, I found that when launched, libreoffice actually uses the class names "soffice" and "Soffice" before switching to the libreoffice class strings identified by xprop.
So i changed my ~/.fluxbox/apps file to:
[group] (workspace=[current])
[app] (name=soffice) (class=Soffice)
[app] (name=libreoffice) (class=libreoffice-writer)
[app] (name=libreoffice) (class=libreoffice-calc)
[Deco] {0x67f}
[Maximized] {yes}
[end]
which is working to group the libreoffice documents and spreadsheets. The libreoffice class designations are no longer needed, but I left them in.
I haven't tested this yet to see if distinctions can be made with soffice-writer, soffice-impress etc ...