AnotherFoxGuy/polyvox
1
0
Fork 0
Code Issues Pull Requests Activity

Stripping out paging from SimpleVolume.

Browse Source
This commit is contained in:
David Williams 2011-04-29 22:16:46 +01:00
parent afbf49f626
commit bc9f1e8ac5
2 changed files with 3 additions and 326 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

126
library/PolyVoxCore/include/SimpleVolume.h
View File

@ -36,110 +36,6 @@ freely, subject to the following restrictions:
namespace PolyVox
{
///The SimpleVolume class provides a memory efficient method of storing voxel data while also allowing fast access and modification.
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
/// A SimpleVolume is essentially a 3D array in which each element (or <i>voxel</i>) is identified by a three dimensional (x,y,z) coordinate.
/// We use the SimpleVolume class to store our data in an efficient way, and it is the input to many of the algorithms (such as the surface
/// extractors) which form the heart of PolyVox. The SimpleVolume class is templatised so that different types of data can be stored within each voxel.
///
/// <b> Basic usage</b>
/// The following code snippet shows how to construct a volume and demonstrates basic usage:
///
/// \code
/// SimpleVolume<Material8> volume(Region(Vector3DInt32(0,0,0), Vector3DInt32(63,127,255)));
/// volume.setVoxelAt(15, 90, 42, Material8(5));
/// std::cout << "Voxel at (15, 90, 42) has value: " << volume.getVoxelAt(15, 90, 42).getMaterial() << std::endl;
/// std::cout << "Width = " << volume.getWidth() << ", Height = " << volume.getHeight() << ", Depth = " << volume.getDepth() << std::endl;
/// \endcode
///
/// In this particular example each voxel in the SimpleVolume is of type 'Material8', as specified by the template parameter. This is one of several
/// predefined voxel types, and it is also possible to define your own. The Material8 type simply holds an integer value where zero represents
/// empty space and any other value represents a solid material.
///
/// The SimpleVolume constructor takes a Region as a parameter. This specifies the valid range of voxels which can be held in the volume, so in this
/// particular case the valid voxel positions are (0,0,0) to (63, 127, 255). Attempts to access voxels outside this range will result is accessing the
/// border value (see getBorderValue() and setBorderValue()). PolyVox also has support for near infinite volumes which will be discussed later.
///
/// Access to individual voxels is provided via the setVoxelAt() and getVoxelAt() member functions. Advanced users may also be interested in
/// the Sampler class for faster read-only access to a large number of voxels.
///
/// Lastly the example prints out some properties of the SimpleVolume. Note that the dimentsions getWidth(), getHeight(), and getDepth() are inclusive, such
/// that the width is 64 when the range of valid x coordinates goes from 0 to 63.
///
/// <b>Data Representaion</b>
/// If stored carelessly, volume data can take up a huge amount of memory. For example, a volume of dimensions 1024x1024x1024 with
/// 1 byte per voxel will require 1GB of memory if stored in an uncompressed form. Natuarally our SimpleVolume class is much more efficient
/// than this and it is worth understanding (at least at a high level) the approach which is used.
///
/// Essentially, the SimpleVolume class stores its data as a collection of blocks. Each of these block is much smaller than the whole volume,
/// for example a typical size might be 32x32x32 voxels (though is is configurable by the user). In this case, a 256x512x1024 volume
/// would contain 8x16x32 = 4096 blocks. The data for each block is stored in a compressed form, which uses only a small amout of
/// memory but it is hard to modify the data. Therefore, before any given voxel can be modified, its corresponding block must be uncompressed.
///
/// The compression and decompression of block is a relatively slow process and so we aim to do this as rarely as possible. In order
/// to achive this, the volume class stores a cache of recently used blocks and their associated uncompressed data. Each time a voxel
/// is touched a timestamp is updated on the corresponding block. When the cache becomes full the block with the oldest timestamp is
/// recompressed and moved out of the cache.
///
/// <b>Achieving high compression rates</b>
/// The compression rates which can be achieved can vary significantly depending the nature of the data you are storing, but you can
/// encourage high compression rates by making your data as homogenous as possible. If you are simply storing a material with each
/// voxel then this will probably happen naturally. Games such as Minecraft which use this approach will typically involve large areas
/// of the same material which will compress down well.
///
/// However, if you are storing density values then you may want to take some care. The advantage of storing smoothly changing values
/// is that you can get smooth surfaces extracted, but storing smoothly changing values inside or outside objects (rather than just
/// on the boundary) does not benefit the surface and is very hard to compress effectively. You may wish to apply some thresholding to
/// your density values to reduce this problem (this threasholding should only be applied to voxels who don't contribute to the surface).
///
/// <b>Paging large volumes</b>
/// The compression scheme described previously will typically allow you to load several billion voxels into a few hundred megabytes of memory,
/// though as explained the exact compression rate is highly dependant on your data. If you have more data than this then PolyVox provides a
/// mechanism by which parts of the volume can be paged out of memory by calling user supplied callback functions. This mechanism allows a
/// potentially unlimited amount of data to be loaded, provided the user is able to take responsibility for storing any data which PolyVox
/// cannot fit in memory, and then returning it back to PolyVox on demand. For example, the user might choose to temporarily store this data
/// on disk or stream it to a remote database.
///
/// You can construct such a SimpleVolume as follows:
///
/// \code
/// void myDataRequiredHandler(const ConstVolumeProxy<MaterialDensityPair44>& volume, const PolyVox::Region& reg)
/// {
/// //This function is being called because part of the data is missing from memory and needs to be supplied. The parameter
/// //'volume' provides access to the volume data, and the parameter 'reg' indicates which region of the volume you need fill.
/// }
///
/// void myDataOverflowHandler(const ConstVolumeProxy<MaterialDensityPair44>& vol, const PolyVox::Region& reg)
/// {
/// //This function is being called because part of the data is about to be removed from memory. The parameter 'volume'
/// //provides access to the volume data, and the parameter 'reg' indicates which region of the volume you need to store.
/// }
///
/// SimpleVolume<Density>volData(&myDataRequiredHandler, &myDataOverflowHandler);
/// \endcode
///
/// Essentially you are providing an extension to the SimpleVolume class - a way for data to be stored once PolyVox has run out of memory for it. Note
/// that you don't actually have to do anything with the data - you could simply decide that once it gets removed from memory it doesn't matter
/// anymore. But you still need to be ready to then provide something to PolyVox (even if it's just default data) in the event that it is requested.
///
/// <b>Cache-aware traversal</b>
/// You might be suprised at just how many cache misses can occur when you traverse the volume in a naive manner. Consider a 1024x1024x1024 volume
/// with blocks of size 32x32x32. And imagine you iterate over this volume with a simple three-level for loop which iterates over x, the y, then z.
/// If you start at position (0,0,0) then ny the time you reach position (1023,0,0) you have touched 1024 voxels along one edge of the volume and
/// have pulled 32 blocks into the cache. By the time you reach (1023,1023,0) you have hit 1024x1024 voxels and pulled 32x32 blocks into the cache.
/// You are now ready to touch voxel (0,0,1) which is right nect to where you started, but unless your cache is at least 32x32 blocks large then this
/// initial block has already been cleared from the cache.
///
/// Ensuring you have a large enough cache size can obviously help the above situation, but you might also consider iterating over the voxels in a
/// different order. For example, if you replace your three-level loop with a six-level loop then you can first process all the voxels between (0,0,0)
/// and (31,31,31), then process all the voxels between (32,0,0) and (63,0,0), and so forth. Using this approach you will have no cache misses even
/// is your cache sise is only one. Of course the logic is more complex, but writing code in such a cache-aware manner may be beneficial in some situations.
///
/// <b>Threading</b>
/// The SimpleVolume class does not make any guarentees about thread safety. You should ensure that all accesses are performed from the same thread.
/// This is true even if you are only reading data from the volume, as concurrently reading from different threads can invalidate the contents
/// of the block cache (amoung other problems).
////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
template <typename VoxelType>
class SimpleVolume
{
@ -230,21 +126,11 @@ namespace PolyVox
uint32_t timestamp;
};
public:
/// Constructor for creating a very large paging volume.
SimpleVolume
(
polyvox_function<void(const ConstVolumeProxy<VoxelType>&, const Region&)> dataRequiredHandler,
polyvox_function<void(const ConstVolumeProxy<VoxelType>&, const Region&)> dataOverflowHandler,
uint16_t uBlockSideLength = 32
);
public:
/// Constructor for creating a fixed size volume.
SimpleVolume
(
const Region& regValid,
polyvox_function<void(const ConstVolumeProxy<VoxelType>&, const Region&)> dataRequiredHandler = 0,
polyvox_function<void(const ConstVolumeProxy<VoxelType>&, const Region&)> dataOverflowHandler = 0,
bool bPagingEnabled = false,
uint16_t uBlockSideLength = 32
);
/// Deprecated constructor - do not use.
@ -280,20 +166,12 @@ namespace PolyVox
void setCompressionEnabled(bool bCompressionEnabled);
/// Sets the number of blocks for which uncompressed data is stored
void setMaxNumberOfUncompressedBlocks(uint16_t uMaxNumberOfUncompressedBlocks);
/// Sets the number of blocks which can be in memory before the paging system starts unloading them
void setMaxNumberOfBlocksInMemory(uint16_t uMaxNumberOfBlocksInMemory);
/// Sets the value used for voxels which are outside the volume
void setBorderValue(const VoxelType& tBorder);
/// Sets the voxel at the position given by <tt>x,y,z</tt> coordinates
bool setVoxelAt(int32_t uXPos, int32_t uYPos, int32_t uZPos, VoxelType tValue);
/// Sets the voxel at the position given by a 3D vector
bool setVoxelAt(const Vector3DInt32& v3dPos, VoxelType tValue);
/// Tries to ensure that the voxels within the specified Region are loaded into memory.
void prefetch(Region regPrefetch);
/// Ensures that any voxels within the specified Region are removed from memory.
void flush(Region regFlush);
/// Removes all voxels from memory
void flushAll();
/// Empties the cache of uncompressed blocks
void clearBlockCache(void);
@ -333,7 +211,6 @@ private:
mutable Vector3DInt32 m_v3dLastAccessedBlockPos;
mutable Block<VoxelType>* m_pLastAccessedBlock;
uint32_t m_uMaxNumberOfUncompressedBlocks;
uint32_t m_uMaxNumberOfBlocksInMemory;
//We don't store an actual Block for the border, just the uncompressed data. This is partly because the border
//block does not have a position (so can't be passed to getUncompressedBlock()) and partly because there's a
@ -355,7 +232,6 @@ private:
float m_fDiagonalLength;
bool m_bCompressionEnabled;
bool m_bPagingEnabled;
};
}

203
library/PolyVoxCore/include/SimpleVolume.inl
View File

@ -36,27 +36,6 @@ freely, subject to the following restrictions:
namespace PolyVox
{
////////////////////////////////////////////////////////////////////////////////
/// When construncting a very large volume you need to be prepared to handle the scenario where there is so much data that PolyVox cannot fit it all in memory. When PolyVox runs out of memory, it identifies the least recently used data and hands it back to the application via a callback function. It is then the responsibility of the application to store this data until PolyVox needs it again (as signalled by another callback function). Please see the SimpleVolume class documentation for a full description of this process and the required function signatures. If you really don't want to handle these events then you can provide null pointers here, in which case the data will be discarded and/or filled with default values.
/// \param dataRequiredHandler The callback function which will be called when PolyVox tries to use data which is not currently in momory.
/// \param dataOverflowHandler The callback function which will be called when PolyVox has too much data and needs to remove some from memory.
/// \param uBlockSideLength The size of the blocks making up the volume. Small blocks will compress/decompress faster, but there will also be more of them meaning voxel access could be slower.
////////////////////////////////////////////////////////////////////////////////
template <typename VoxelType>
SimpleVolume<VoxelType>::SimpleVolume
(
polyvox_function<void(const ConstVolumeProxy<VoxelType>&, const Region&)> dataRequiredHandler,
polyvox_function<void(const ConstVolumeProxy<VoxelType>&, const Region&)> dataOverflowHandler,
uint16_t uBlockSideLength
)
{
m_funcDataRequiredHandler = dataRequiredHandler;
m_funcDataOverflowHandler = dataOverflowHandler;
m_bPagingEnabled = true;
//Create a volume of the right size.
resize(Region::MaxRegion,uBlockSideLength);
}
////////////////////////////////////////////////////////////////////////////////
/// Deprecated - do not use this constructor.
////////////////////////////////////////////////////////////////////////////////
@ -90,27 +69,19 @@ namespace PolyVox
SimpleVolume<VoxelType>::SimpleVolume
(
const Region& regValid,
polyvox_function<void(const ConstVolumeProxy<VoxelType>&, const Region&)> dataRequiredHandler,
polyvox_function<void(const ConstVolumeProxy<VoxelType>&, const Region&)> dataOverflowHandler,
bool bPagingEnabled,
uint16_t uBlockSideLength
)
{
m_funcDataRequiredHandler = dataRequiredHandler;
m_funcDataOverflowHandler = dataOverflowHandler;
m_bPagingEnabled = bPagingEnabled;
//Create a volume of the right size.
resize(regValid,uBlockSideLength);
}
////////////////////////////////////////////////////////////////////////////////
/// Destroys the volume The destructor will call flushAll() to ensure that a paging volume has the chance to save it's data via the dataOverflowHandler() if desired.
/// Destroys the volume
////////////////////////////////////////////////////////////////////////////////
template <typename VoxelType>
SimpleVolume<VoxelType>::~SimpleVolume()
{
flushAll();
}
////////////////////////////////////////////////////////////////////////////////
@ -274,20 +245,6 @@ namespace PolyVox
m_uMaxNumberOfUncompressedBlocks = uMaxNumberOfUncompressedBlocks;
}
////////////////////////////////////////////////////////////////////////////////
/// Increasing the number of blocks in memory causes fewer calls to dataRequiredHandler()/dataOverflowHandler()
/// \param uMaxBlocks The number of blocks
////////////////////////////////////////////////////////////////////////////////
template <typename VoxelType>
void SimpleVolume<VoxelType>::setMaxNumberOfBlocksInMemory(uint16_t uMaxNumberOfBlocksInMemory)
{
if(m_pBlocks.size() > uMaxNumberOfBlocksInMemory)
{
flushAll();
}
m_uMaxNumberOfBlocksInMemory = uMaxNumberOfBlocksInMemory;
}
////////////////////////////////////////////////////////////////////////////////
/// \param tBorder The value to use for voxels outside the volume.
////////////////////////////////////////////////////////////////////////////////
@ -338,122 +295,6 @@ namespace PolyVox
return setVoxelAt(v3dPos.getX(), v3dPos.getY(), v3dPos.getZ(), tValue);
}
////////////////////////////////////////////////////////////////////////////////
/// Note that if MaxNumberOfBlocksInMemory is not large enough to support the region this function will only load part of the region. In this case it is undefined which parts will actually be loaded. If all the voxels in the given region are already loaded, this function will not do anything. Other voxels might be unloaded to make space for the new voxels.
/// \param regPrefetch The Region of voxels to prefetch into memory.
////////////////////////////////////////////////////////////////////////////////
template <typename VoxelType>
void SimpleVolume<VoxelType>::prefetch(Region regPrefetch)
{
Vector3DInt32 v3dStart;
for(int i = 0; i < 3; i++)
{
v3dStart.setElement(i, regPrefetch.getLowerCorner().getElement(i) >> m_uBlockSideLengthPower);
}
Vector3DInt32 v3dEnd;
for(int i = 0; i < 3; i++)
{
v3dEnd.setElement(i, regPrefetch.getUpperCorner().getElement(i) >> m_uBlockSideLengthPower);
}
Vector3DInt32 v3dSize = v3dEnd - v3dStart + Vector3DInt32(1,1,1);
int32_t numblocks = v3dSize.getX() * v3dSize.getY() * v3dSize.getZ();
if(numblocks > m_uMaxNumberOfBlocksInMemory)
{
// cannot support the amount of blocks... so only load the maximum possible
numblocks = m_uMaxNumberOfBlocksInMemory;
}
for(int32_t x = v3dStart.getX(); x <= v3dEnd.getX(); x++)
{
for(int32_t y = v3dStart.getY(); y <= v3dEnd.getY(); y++)
{
for(int32_t z = v3dStart.getZ(); z <= v3dEnd.getZ(); z++)
{
Vector3DInt32 pos(x,y,z);
typename std::map<Vector3DInt32, LoadedBlock>::iterator itBlock = m_pBlocks.find(pos);
if(itBlock != m_pBlocks.end())
{
// If the block is already loaded then we don't load it again. This means it does not get uncompressed,
// whereas if we were to call getUncompressedBlock() regardless then it would also get uncompressed.
// This might be nice, but on the prefetch region could be bigger than the uncompressed cache size.
// This would limit the amount of prefetching we could do.
continue;
}
if(numblocks == 0)
{
// Loading any more blocks would attempt to overflow the memory and therefore erase blocks
// we loaded in the beginning. This wouldn't cause logic problems but would be wasteful.
return;
}
// load a block
numblocks--;
Block<VoxelType>* block = getUncompressedBlock(x,y,z);
} // for z
} // for y
} // for x
}
////////////////////////////////////////////////////////////////////////////////
/// Removes all voxels from memory, and calls dataOverflowHandler() to ensure the application has a chance to store the data.
////////////////////////////////////////////////////////////////////////////////
template <typename VoxelType>
void SimpleVolume<VoxelType>::flushAll()
{
typename std::map<Vector3DInt32, LoadedBlock >::iterator i;
//Replaced the for loop here as the call to
//eraseBlock was invalidating the iterator.
while(m_pBlocks.size() > 0)
{
eraseBlock(m_pBlocks.begin());
}
}
////////////////////////////////////////////////////////////////////////////////
/// Removes all voxels in the specified Region from memory, and calls dataOverflowHandler() to ensure the application has a chance to store the data. It is possible that there are no voxels loaded in the Region, in which case the function will have no effect.
////////////////////////////////////////////////////////////////////////////////
template <typename VoxelType>
void SimpleVolume<VoxelType>::flush(Region regFlush)
{
Vector3DInt32 v3dStart;
for(int i = 0; i < 3; i++)
{
v3dStart.setElement(i, regFlush.getLowerCorner().getElement(i) >> m_uBlockSideLengthPower);
}
Vector3DInt32 v3dEnd;
for(int i = 0; i < 3; i++)
{
v3dEnd.setElement(i, regFlush.getUpperCorner().getElement(i) >> m_uBlockSideLengthPower);
}
for(int32_t x = v3dStart.getX(); x <= v3dEnd.getX(); x++)
{
for(int32_t y = v3dStart.getY(); y <= v3dEnd.getY(); y++)
{
for(int32_t z = v3dStart.getZ(); z <= v3dEnd.getZ(); z++)
{
Vector3DInt32 pos(x,y,z);
typename std::map<Vector3DInt32, LoadedBlock>::iterator itBlock = m_pBlocks.find(pos);
if(itBlock == m_pBlocks.end())
{
// not loaded, not unloading
continue;
}
eraseBlock(itBlock);
// eraseBlock might cause a call to getUncompressedBlock, which again sets m_pLastAccessedBlock
if(m_v3dLastAccessedBlockPos == pos)
{
m_pLastAccessedBlock = 0;
}
} // for z
} // for y
} // for x
}
////////////////////////////////////////////////////////////////////////////////
///
////////////////////////////////////////////////////////////////////////////////
@ -490,7 +331,6 @@ namespace PolyVox
m_uMaxNumberOfUncompressedBlocks = 16;
m_uBlockSideLength = uBlockSideLength;
m_pUncompressedBorderData = 0;
m_uMaxNumberOfBlocksInMemory = 1024;
m_v3dLastAccessedBlockPos = Vector3DInt32(0,0,0); //There are no invalid positions, but initially the m_pLastAccessedBlock pointer will be null;
m_pLastAccessedBlock = 0;
m_bCompressionEnabled = true;
@ -597,49 +437,10 @@ namespace PolyVox
typename std::map<Vector3DInt32, LoadedBlock >::iterator itBlock = m_pBlocks.find(v3dBlockPos);
// check whether the block is already loaded
if(itBlock == m_pBlocks.end())
{
//The block is not in the map, so we will have to create a new block and add it.
//Before we do so, we might want to dump some existing data to make space. We
//Only do this if paging is enabled.
/*if(m_bPagingEnabled)
{
// check wether another block needs to be unloaded before this one can be loaded
if(m_pBlocks.size() == m_uMaxNumberOfBlocksInMemory)
{
// find the least recently used block
typename std::map<Vector3DInt32, LoadedBlock >::iterator i;
typename std::map<Vector3DInt32, LoadedBlock >::iterator itUnloadBlock = m_pBlocks.begin();
for(i = m_pBlocks.begin(); i != m_pBlocks.end(); i++)
{
if(i->second.timestamp < itUnloadBlock->second.timestamp)
{
itUnloadBlock = i;
}
}
eraseBlock(itUnloadBlock);
}
}*/
{
// create the new block
LoadedBlock newBlock(m_uBlockSideLength);
itBlock = m_pBlocks.insert(std::make_pair(v3dBlockPos, newBlock)).first;
//We have created the new block. If paging is enabled it should be used to
//fill in the required data. Otherwise it is just left in the default state.
/*if(m_bPagingEnabled)
{
if(m_funcDataRequiredHandler)
{
// "load" will actually call setVoxel, which will in turn call this function again but the block will be found
// so this if(itBlock == m_pBlocks.end()) never is entered
//FIXME - can we pass the block around so that we don't have to find it again when we recursively call this function?
Vector3DInt32 v3dLower(v3dBlockPos.getX() << m_uBlockSideLengthPower, v3dBlockPos.getY() << m_uBlockSideLengthPower, v3dBlockPos.getZ() << m_uBlockSideLengthPower);
Vector3DInt32 v3dUpper = v3dLower + Vector3DInt32(m_uBlockSideLength-1, m_uBlockSideLength-1, m_uBlockSideLength-1);
Region reg(v3dLower, v3dUpper);
ConstVolumeProxy<VoxelType> ConstVolumeProxy(*this, reg);
m_funcDataRequiredHandler(ConstVolumeProxy, reg);
}
}*/
}
//Get the block and mark that we accessed it