6. Large outputs: the index

To post-process its results, solver does not parse results.xml into a DOM: it indexes it. An xml_index holds the tags and their attributes, but no content: the memory it needs does not depend on the data in the file.

program solver
!< Tutorial, chapter 6: large outputs, the index.
use foxy
implicit none
type(xml_index)           :: results
integer(I4P), allocatable :: snapshots(:)
real(R8P),    allocatable :: pressure(:)
integer(I4P)              :: s, iostat, step
integer(I8P)              :: first, last

call results%scan(filename='results.xml', iostat=iostat)   ! tags and attributes only, no content
snapshots = results%find_all(path='results/snapshot')
print '(A,I0,A,I0,A)', 'indexed ', results%tags_number(), ' tags, ', size(snapshots), ' snapshots'
do s=1, size(snapshots)
   call results%get_attribute(id=snapshots(s), name='step', value=step)
   call results%get_content(id=results%find(name='pressure', parent=snapshots(s)), values=pressure)  ! read now
   call results%content_range(id=results%find(name='pressure', parent=snapshots(s)), first=first, last=last)
   print '(A,I3,A,3F7.3,A,I0,A,I0)', 'step ', step, ': p =', pressure, ', bytes ', first, '-', last
enddo
call results%scan(filename='results.xml', stop_at='pressure')   ! stop after the first <pressure ...>
print '(A,I0,A,I0)', 'stopped at byte ', results%stop_position(), ', tags indexed: ', results%tags_number()
endprogram solver

• scan reads the file in chunks and records every tag: name, attributes, hierarchy and byte positions.
• The queries are the ones of xml_file: find, find_all, children, attribute, get_attribute, ...
• content(id) and get_content(id, ...) read the content of one tag from the file, when you need it.
• content_range gives the bytes between the start and the end tag of a tag (its children included), text_range only its direct text, up to its first child: read them yourself, e.g. a binary block.
• scan(stop_at=) stops after the first start tag with that name: what follows (e.g. the raw appended data of a VTK file) is not parsed, and nothing is read beyond the current chunk (64 KiB by default). stop_position() is the first byte after that start tag.
• The index reads the file bytes (its positions are file positions): a UTF-16 or UTF-32 file is refused; parse those with xml_file or a handler, which convert them.

Running it

$ solver
indexed 10 tags, 3 snapshots
step 100: p =  1.000  0.500  0.333, bytes 190-206
step 300: p =  0.333  0.167  0.111, bytes 347-363
step 500: p =  0.200  0.100  0.067, bytes 504-520
stopped at byte 190, tags indexed: 4

What you learned

Index large files with xml_index and read only the contents you need. Reference: The index.

Next: 7. Your own handler.