Binary woes.

Did you know if you mix binary reads with normal text reads that eventually the reads will fail?  I didn't know until today and it took a while to track down the root cause.  All of my IO up to this point has been simple fputs/fputc/fgetc which has worked fine, but with the level loading I wanted to serialized the tiles array (basically the level layout) as a binary chunk in the midst of a normal text file.  Saving that worked as expected, where I end up with something like the following...

object Level_0 type=Level
  TileBinaryData:{insert lots of random bits here}
  Width=50
  Height=50
  LevelName="TestLevel"
end

I added a couple hooks to my Reader/Writer classes that wrap around fread/fwrite which just takes a pointer and a size essentially.  Writing worked fine as I said, but about ~400 bytes into reading the tiles some internal buffer in the FILE structure would run out, and fread would start to fail from then on.  That led me to believe there's a fixed limit on how big text files will successfully read but that's not the case as my binary data actually reduces the file size quite a bit as I'm able to be more efficient with the storage.

At any rate, adding a little 'b' to the fopen calls when appropriate seems to have solved the issues just fine.  Not a big deal once I figured it out but it was a bit frustrating to come across it in this manner.  Looks like this isn't terribly uncommon either as a quick google search yielded a few hits...

http://stackoverflow.com/questions/474733/unexpected-output-copying-file-in-c

File IO seems like exactly the kind of API that is unforgiving, somewhat finicky to get working right, and once you're done you never touch it again.  I look forward to never having to touch it again some day.  As an upside level saving/loading is now significantly faster, and there's even further room to optimize as I add binary import/export options to the property system (currently only doing manual binary writes for the tiles).

Great success!

Level saving and loading works!  I need to fix up a few more things and optimize some obvious bits, but the core functionality is there!  I would post a screenshot to show the success but it really wouldn't look any different than a normal shot - oh well.

I'm currently exporting out all level objects in a text format (the same format I've built for loading data) which seems to work fairly well.  It's fairly slow to parse (string manipulation in C++ isn't really a pleasant process, and I'm sure my string class implementation isn't helping matters much) and it takes quite a bit of space (current 50x50 level with 15 creatures is taking about 185kb on disk), but the upside is that it's human readable/editable/debuggable.

The basic process for saving is as follows:

- Construct a new package object
- Add current level to the root set of objects for that package
- Copy that root set to a new set of ObjectsToSave
- Start traversing all objects in the ObjectsToSave set, adding any objects they reference to ObjectsToSave such that we recurse through the entire object "tree"
- Write out a "packageinfo" block, which is really just forward declarations of objects contained in the package (useful for loading which I'll point out below)
- Iterate through the ObjectsToSave set and write out any data that differs from the current set of defaults

Fortunately I'm able to leverage the object/property system pretty well for this, so at the highest logical point it's only 15 lines of code or so to save a set of objects.  Loading is a little more complicated at the top level since I have to do some extra parsing to know what data is incoming, but it's still pretty manageable at this point.  At any rate, the process for loading is currently:

- Construct a new package object
- Open the package, read the "packageinfo" which will create all of the objects contained in this package using the default values and creates them using the same names that they had on save.  This allows us to easily find references to other objects contained in the packages when loading without having to resort to a separate fix-up phase.
- Read in the objects' data
- Hand out some post-load notifications for specific objects to fix-up any data (currently only the level object does some munging)

There's also a little trickery of finding the loaded player, copying over it's values to the current player, and then deleting the loaded version.  All in all it's fairly straightforward and seems to work fairly well.  It took quite a while to find all the properties that I either wasn't saving out that I needed to, or all the other bits of data that get initialized on level creation and now needs to be handled separately.  For the most part this has helped me find older chunks of code that needed a slight refactoring anyways which is always a good thing (although I did wonder if it would ever end on several occasions).

Woot!

Good design reference

Listing this more so that I don't forget about it, but someone else may find it useful -

http://www.designersnotebook.com/Design_Resources/No_Twinkie_Database/no_twinkie_database.htm

Bit in the ass...

Ran into this the hard way, and the first hit on google clearly explains the issue:
http://www.artima.com/cppsource/nevercall.html.  I can't wait until I finish this project and move on to another language with a different (hopefully more obvious) set of quirks.

Hey, that ain't right.

Still plugging away, although some friends at work convinced me to play some WoW again to prepare for the impending goblin invasion, so progress has been a little stilted.  At any rate, I've managed to make some headway on a few important additions, the easiest and yet most work creating of which was writing a proper logging system.

In the past I only had in-game logging meant for game messages that I was hijacking for specific errors/warnings I cared about (syntax errors when loading data files mostly).  Given that I didn't want to flood the game UI with a bunch of other information it tended to be very specific and not terribly helpful - but now I've added a log which goes straight to disk, which has given me freedom to spread little Logf()'s all throughout my code base.  Adding some to a few key places exposed some interesting bugs that would have surely gone undetected for quite some time.

- All objects were doing duplicate work from a copy and paste error in my object definition macros (gg virtual destructors, you win).
- Level objects were being added to the tickable list twice which caused double game updates.  I don't have much in game animation yet, so it's not really noticeable at this stage but it would have been annoying once I started implementing projectiles and other effects.
- Object instance counts were incrementing on one type (correct) and decrementing on another (incorrect) for dynamic types.  Fortunately the right counter was being incremented, so I wasn't seeing any funky object naming (or name clashing) yet, but again this would inevitably cause pain at some point.

Aside from that fun I've written most of the framework I think I'll need to save level instances.  I've already added the logic to link up levels correctly to each other (stairs up match the stairs down on the other side for example) and left the hooks needed to handle other entities transitioning across levels once I get to that feature.  Hopefully in another 100 lines of code or so this game will finally be susceptible to save scumming!

Updates

I've made a few additions over the last week or so, figured it's worth a quick post to list some of the bigger points.

Zones now exist and act as collections for levels, storing the information needed to generate them as needed, along with the information to link them together.   I've worked through the basic functionality to create levels and transition between them via LevelLinks (aka stairs, hatches, etc) through player interaction.  Along with this came several fixes related to cleaning up Levels for deletion and it looks like there's definitely more work to be done here, but for now the basic framework is in place and working.  It was a slightly magical moment the first time I worked out the last kinks and was able to travel back and forth between levels repeatedly...

As I mentioned in the previous post, entering the data for the starting zone exposed some limitations of my data file parsing system.  So I ended up spending a fair bit of time shoring this up and cleaning up some annoyances that had been bothering me for a while.  First and foremost was simplifying the declaration of properties for serialization to make it less error prone and more readable.

The old way (with lots of snipping):

REGISTER_TYPE(ACreatureClass)
void ACreatureClass::DefineProperties(ObjectType *Type, Property *LastProperty)
{
ADD_PROPERTY(CREATE_PROPERTY_STRUCT(ACreatureClass,TextCell,Cell));
ADD_PROPERTY(CREATE_PROPERTY_STRUCT(ACreatureClass,Range_Int,BaseHealth));
ADD_PROPERTY(CREATE_PROPERTY(ACreatureClass,float,BaseEvasion));
ADD_PROPERTY(CREATE_PROPERTY(ACreatureClass,int,BaseMovementSpeed));
ADD_PROPERTY(CREATE_PROPERTY_CUSTOM(ACreatureClass,String,BrainTypeName));
ADD_PROPERTY(CREATE_PROPERTY(ACreatureClass,int,BaseSightRadius));
ADD_PROPERTY(CREATE_PROPERTY_OBJECT(ACreatureClass,AWeapon,StartingWeapon));
ADD_PROPERTY(CREATE_PROPERTY_ARRAY_EXT(ACreatureClass,ItemBodySlotDesc,BodySlots,CREATE_PROPERTY_STRUCT(ACreatureClass,ItemBodySlotDesc,BodySlots)));
}

And the new way:

REGISTER_TYPE(ACreatureClass)
CREATE_PROPERTY_STRUCT(TextCell,Cell)
CREATE_PROPERTY_STRUCT(Range_Int,BaseHealth)
CREATE_PROPERTY(float,BaseEvasion)
CREATE_PROPERTY(int,BaseMovementSpeed)
CREATE_PROPERTY_STRING(BrainTypeName)
CREATE_PROPERTY(int,BaseSightRadius)
CREATE_PROPERTY_OBJECT(AWeapon,StartingWeapon)
CREATE_PROPERTY_ARRAY_STRUCT(ItemBodySlotDesc,BodySlots)
END_REGISTER_TYPE(ACreatureClass)

I've also added a property type for referencing data types, which will make a safer replacement for the string lookups I've been using (BranTypeName, ClassName, etc).  Array properties also received some bolstering, making it possible to access properties by index and support for referencing inline other property definitions which comes in handy when you just need to touch one property of an object stored in base type's array.

At the moment I'm focusing on getting saving to disk working so I can have some sort of persistence in this game.  So far I've written a basic file writer and a debug command to test the property system; the remaining work doesn't seem that far off now.  Looks like the biggest problem to solve next is how to convert pointers into some sort of lookup table that I can safely save off and then rebuild on load when needed.

Working on the first zone...

I decided a good way to gain traction is to set a pretty clear (and attainable) goal - ideally something that helps progress the "game" side of this project rather than just more engine tweaks.  To that end I've started piecing together the basics of a multi-level dungeon with a simple progression of environment, creatures, and at some point loot.  Thus begins the first goblin caves adventure...

Of course I ran into a bit of a snag when I started writing the class definition for zones, specifically trying to lock down what a zone needs to know and how best to represent it.  Ultimately I decided to take the approach of writing the zone data in whatever way seemed natural, and then worked backwards to turn that into C++ definitions.  That of course exposed some limitations of my data file format/parsing that I'll definitely want to revisit soon, but for now it'll get the job done.

I'm Alive! (no really!)

Finally getting settled, started actually writing a bit of code again (Final Fantasy XIII delayed this process by a couple weeks).  It's hard to keep myself focused though - I've been toying with trying a new source depository ala github (or unfuddle), and then tried setting up TFS at home since it integrates (supposedly) nicely with Visual Studio 2010.  Meanwhile not a whole lot of code is actually being written, but I did manage to improve the SDL port a bit.

Instead of trying to alpha blend in various colors to achieve the sprite re-coloring I wanted, I ended up switching to a much simpler palette swapping approach.  At some point I need to refactor how the UI handles sizing/positioning to account for the SDL pixel based resolutions rather than the console's column/row resolution.  Ideally I'll put SDL aside again for a while and focus on more game features using the basic console since it's working well.  Procrastination is a horrible behavior that I'm really good at unfortunately!

At any rate I found this blurb pretty amusing.. http://www.dadhacker.com/blog/?p=1231

Moving!

Not a whole lot of progress is likely in the next couple of weeks since my wife and I are moving across country to Seattle.  North Carolina has been a great place to live, but we're looking for a change of location (especially to somewhere with mountains nearby), and it's a good excuse to mix things up career-wise as well.  Hopefully I'll get my main dev machine back from the movers in a week or so, and in the meantime I've got my laptop which will have to do for now.

In the meantime go check out the second Magecrawl Tech demo.

SDLconsole support

I decided to dust off the SDL wrapper for my console class to see how it's held up.  Somehow my switch to unicode support introduced a nice obscure link error that took at while to track down - project settings were correct, just needed to include sdlmain.lib now explicitly.  The wrapper was sort of half implemented and there was no useful keyboard support other than up/down/enter, so I went ahead and plugged in some basics so none of the menus are blocked.

It will need quite a bit of work to make it pretty that's for sure, but the basics are there.  Extended ascii characters in the font for one so that my grass and walls show up would be nice, and then some sort of method to expose true 32bit color support for SDL but still be able to bucket it for the console builds.  I have a couple ideas on how to have the property system automatically support this extra data without too much trouble, and then it would be a matter of just making sure there's plenty of fallbacks when the extra data in unnecessary/unwanted.

Progress

I've knocked out some major steps that makes it likely that this project might actually playable at some point.  I've updated the picasa gallery with 5 new screenshots, check the entire album out here.  I'm nearly up to 150 submissions in my source repository so far, spanning over 2 years (with huge gaps, looks like I didn't touch the project for a year and a half altogether), and it's finally starting to get some legs.  Lots of work ahead of me no doubt, but I'm excited to have made it this far.

• Inventory is now implemented in a mostly usable form.  Creatures have a configurable list of body slots that Items can be equipped too, along with all the appropriate hooks for status modification down the road (cursed items, status changes, etc).  I spent quite a bit of time on the inventory screen as well, which worked well in testing out both the inventory and UI frameworks.

• Combat has been scrubbed a bit as I try to figure out how exactly the mechanics of attacking, dodging, and damage absorption should work.  I've gone and forth on a couple systems, but for now I think I've settled on the idea that items themselves hold the most influence on combat values, while creature stats and skills either scale those values or enforce a minimum.  Weapons for example dictate a base chance to hit, attack speed, and damage range, and a player's agility would modify the minimum chance to hit and attack speeds, while the matching weapon skill would scale the weapon's base values.  I have no idea if this approach will work or not in the long term, but for now I've got something in that I can start playing with.

• It's now possible to level up!  Coming up with an XP table generator is voodoo magic, and I ended up spending an hour or so playing with some weird combination of formulas that eventually generated something that looked "right".  The more important part is that it's now possible to go past level 1, so there's actually a point to killing all those goblins and rats.

• Level Generator and Populator classes are now roughly implemented, and I moved my basic cave generator into it's new (data driven) home.  Now it's trivial to tweak all the various parameters of the generation or creature population, and no longer requires recompiling the executable.  I think the growing pains with my property reflection/serialization system are starting to yield some awesome rewards.  It's very rewarding to watch all the individual components I've been working start to come together.  This shot shows the debug console with some of my in-game debug commands to inspect the  type registry, and then dive into the properties of a specific type/instace.

I'm looking forward to start adding more and more of the actual game bits now that the underlying frameworks are mostly working.  I can tell there's going to be more fun ahead already... I'm pretty sure my current implementation of time in the engine is going to be asking for a rewrite very shortly.

Macros really are evil... :(

I knew my property system macros were fragile, and I ran into a fun one the other day when adding properties for the new level generators.  When loading up some test data my structure was getting turned into garbage, which I eventually tracked down to the second property in the struct having an invalid offset.  Thankfully Visual Studio's debugger is badass, and was able to find that I had accidentally setup a string property (12 bytes) into a string pointer (4 bytes), doh!

struct LevelPopulatorCreatureInfo
{
    TString* CreatureName;   // this shouldn't be a pointer, and throws off all other offsets by 8
    ...
}

ADD_PROPERTY(CREATE_PROPERTY_CUSTOM(LevelPopulatorCreatureInfo,String,CreatureName));

The question now is to add some verification asserts to the already convoluted macros, or should I bite the bullet and refactor the system to make the compiler do the type checking?  This might be a fun post to look back on in a couple months....

Macros are evil...

For some reason I thought it would be clever (and elegant) to wrap my property system declarations in some simple macros, which of course worked decently for simple cases...

DEFINE_PROPERTY_STRUCT(ACreatureClass,Range_Int,BaseXpValue));

DEFINE_PROPERTY(ACreatureClass,float,BaseEvasion));

The macro handles creating a new Property class instance and adding it to the DataType's list of properties, etc etc. Now recently I find myself add a new Property_Array class (now that I'm actually creating a game I have need of data driven lists), which worked out fairly well initially

#define DEFINE_PROPERTY_ARRAY(ClassType,PropType,PropName) Type->Properties.Add( new Property_Array(#PropName,GET_OFFSET(ClassType,PropName)) );

Which worked great for my test case for array of integers, but quickly fell apart once I wanted to make an array of enums (or for more pain, an array of structs of arrays...). I've since refactored the macros to support embedding of property creation within the array property creation macro, but it's turned into a real mess that I'm not proud of...

ADD_PROPERTY(CREATE_PROPERTY_ARRAY_EXT(ACreatureClass,ItemBodySlot,BodySlots,CREATE_PROPERTY_ENUM(ACreatureClass,ItemBodySlot,BodySlots,NumItemBodySlots,ItemBodySlots,ItemBodySlotNames)));

And with the (ab)use of the temporary LastProperty it's a pretty frail system, where any bit of manual monkeying when defining properties could very easy cause bad things. Alas I've resisted the temptation to refactor the entire system altogether for now (got to make a game you see!), but at least there's a nice juicy comment admitting the problems with the approach. As if that somehow makes it okay to write bad code....