Rewrote the build system. Faster, build/assets/, etc.

[Icedude907]

This rewrites most of the build system. Yep.
NOTE: I renamed the branch on my system and the PR self-destructed. See here: https://github.com/Icedude907/pokeemerald/tree/build_old

Description

Main changes:

• Rewrote, renamed, and partially documented most makefiles. Numerous internal changes - the main rules remain the same.
  Improved buildspeed (by making changes to scaninc)

• Moved all generated assets into a build/assets folder. Includes:
  .gbapal .1bpp .4bpp .8bpp .lz .rl .s(midi) .bin(audio)

• Moved all generated .inc and .h files into build/assets/data and build/assets/include/generated respectively.
  This required changes to mapjson to support setting output directories.

• All midi files now have an associated .cfg file containing the mid2agb options they use.
  It might be worth condensing all these separate files into one, but I'm not sure.

• Removed all non-tool .gitignore files except the root one (and tidied it up a bit)

• Removed deprecated build_tools.sh

• Removed an accidentally committed redyellowgreen_frame.bin

• Renamed japanese_frlg_male_font.png to japanese_frlg_male.png and same for female.

• Moved spinda_spots inside the Pokemon's folder

• Renamed the ld_*.txt files to ld_*.ld

Future changes I'd like to see

• Move most buildscripts out of the root folder
• Remove extra mkdirs
• Config files in place of most of spritesheet_rules.mk & graphics_rules.mk
• Allow changing the build directory (Few problems, e.g.: preproc doesn't expand macros inside INCBIN)

ETC

• Closes Some build artifacts are not in build/ #1224
• make clean no longer removes tools (use make clean-all or clean-tools)
• You should make clean before opening this branch.
  (If you don't you can use make clean-assets-old to remove the majority of the old files)
• I've never seriously looked at Make prior to this, I'll have missed few things I missed.

Questions (assistance please)

• How is NODEP supposed to work? I'm not sure I've properly preserved its behaviour.
• Why are there special compilation options for some files in the modern build?

Discord contact info

Idk why you need this... 'icedude907'

Addenda

It seems that certain portions of this PR could break a lot of downstream tools. I'm willing to subdivide it, but for now I'd appreciate general feedback.

The rationale for the .cfg change is that new users (like me) adding additional songs may comprehend the process quicker than digging through the build system for a rule - admittedly, a guide would serve much the same purpose.

[Jaizu]

I will let more technical/savvy people judge the quality of the changes but I think we should consider current projects when making big PRs like this. This will be conflict hell, specially for newer users.
Should we try to approach this with different PRs that do their own thing instead making a big PR that tries to acomplish lots of different things? Doing smaller PRs will help people have a better time merging something as big as this.

[mrgriffin]

I like the idea of being able to make clean by simply rm -rf build. (Morally speaking, I know that's not literally what's done)

Since it seems that assets are only ever built in build/assets, does that mean that INCBINs only ever include from that directory, and thus could implicitly be relative to that directory? e.g. make INCBIN_U16("foo.gbapal") include the contents of build/assets/foo.gbapal. I think that could make it easier for downstreams to adapt to this change (because they wouldn't have to update all their INCBINs, nor deal with conflicts from having edited code near existing INCBINs [because we wouldn't be updating all the vanilla INCBINs])?

(Few problems, e.g.: preproc doesn't expand macros inside INCBIN)

Haha, yeah, on my list of things to try is implementing a cpp which understands INCBIN so that it's able to expand macros. Also something like that would be handy for scaninc, because currently if you write:

#if SOME_CONDITION
const u8 gFoo = INCBIN_U8("foo");
#else
const u8 gFoo = INCBIN_U8("bar");
#endif

scaninc incorrectly decides that the file depends on both foo and bar. (This comes up when people want to use the preprocessor to choose between graphics, think something like being able to toggle gen 3 vs gen 4 battle sprites).

[GriffinRichards]

does that mean that INCBINs only ever include from that directory, and thus could implicitly be relative to that directory?

All the uncompressed .bin files for example are still included from outside the build folder. I agree though that I would prefer this not edit all these paths. INCBIN should really work the same way include does: checking the listed path for the file, then build/assets/<path> if it doesn't exist.

[GriffinRichards]

The build fails on GitHub. It looks like you're still not generating include/constants/map_groups.h and layouts.h before they're needed.

It fails for me locally as well, because you've bumped the version requirements by using <filesystem>

[Icedude907]

It looks like you're still not generating include/constants/map_groups.h and layouts.h before they're needed.

Curious. This is to do with Ubuntu bundling an old version of make. Hoping I've fixed it now.

It fails for me locally as well, because you've bumped the version requirements by using

Really? jsonproc was already building with C++17 last year, so I assumed that'd be ok. See here.
I could remove the dependency, but then there'd be a few more mkdir calls around.

[Icedude907]

Haha, yeah, on my list of things to try is implementing a cpp which understands INCBIN so that it's able to expand macros.

I wonder about INCBIN.
I think modern assemblers support an .incbin directive (which can then be used in C via an asm macro) and we could probably rig agbcc to do the same - this might be a simpler and more efficient solution to the problem, with include paths just working as an added side-effect.

[GriffinRichards]

Really? jsonproc was already building with C++17 last year, so I assumed that'd be ok.

Yeah, for macOS the standard library features have lagged behind the language. I assume jsonproc (or any other part of the repo) wasn't previously using anything introduced to the library in C++17.

A few extra mkdir calls is ok, I'd rather that than lose support for some older compilers.

[Icedude907]

I didn't realise this would close itself.
Anyway, I'm going to have another go, any requests as how you'd like my PR(s) to be formatted?