Create configurable base year - so we can start from 2010 or 2020 and achieve the same result #384

lmz · 2024-10-17T19:30:43Z

This PR includes changes so that BAUS can be started in 2010 or 2020 and the results will be the same. This is achieved by:

Running a new debug model as the first simulation model. When 2020 is not the base_year but the model year is 2020, this was used to write an h5 store with 2020 snapshots of the input tables. This was used to create the input table, 2024_10_29_bayarea_2020start.h5
Upgrade caching. A bunch of columns were set to cache=True, and weren't being updated when the tables were updated. These were modified to have cache_scope='step'. This also requires using a local version of urbansim_defaults: https://github.com/BayAreaMetro/urbansim_defaults/commits/master/
Some small changes for compatibility:
a. scheduled_development_events(): Convert special case from base_year + years_per_iter to 2010 + years_per_iter
b. scheduled_development_events(): Don't add parcels to static_parcels until 2020.
c. When base_year is 2020:
1. store() reads 2024_10_29_bayarea_2020start.h5 rather than 2015_09_01_bayarea_v3.h5
2. lump_sum_accounts(): Add 2015 lump sum to account to be consistent with 2010 start. (I think this can be removed).

Other changes:

Made program argument, --run_setup_yaml, so multiple versions of BAUS can be tested simultaneously with the same code base via different run setup files.
Add base_year and final_year into the run_setup_yaml.
Remove run_name prefix from output filenames. This way output directories between runs can be diffed with other run output directories. The run_name prefix is still present when copied into the visualization directory.
Rename some output files for clarity: disaggregate buildings, households, jobs, etc files that aren't summarized to a geography are called XX_table.csv rather than XX_summary.csv
Send update message to slack at the end of each iteration
Fix warnings by replacing yaml.load() with yaml.safe_load()
Convert static_parcels list in configs/developer/developer_settings.yaml to parcel ids rather than geom_ids
Add column, move_in_year, for households and jobs to understand when they located
Debugging added to https://github.com/BayAreaMetro/urbansim/commits/master/ -- use this

To see the specific tasks where the Asana app for GitHub is being used, see below:
- https://app.asana.com/0/0/1207556028226924

…with the same code

From M:\urban_modeling\baus\PBA50Plus\PBA50Plus_DraftBlueprint\PBA50Plus_Draft_Blueprint_v8_znupd_nodevfix\run_setup_PBA50Plus_Draft_Blueprint_v8_znupd_nodevfix.yaml

This commit reproduces PBA50+ DBP run

(baus-env-2023) E:\LUModel2Share\GitHub\bayarea_urbansim>git diff M:\urban_modeling\baus\PBA50Plus\PBA50Plus_DraftBlueprint\PBA50Plus_Draft_Blueprint_v8_znupd_nodevfix\geographic_summaries\PBA50Plus_Draft_Blueprint_v8_znupd_nodevfix_county_summary_2025.csv E:\LUModel2Share\PBA50Plus_DraftBlueprint\PBA50Plus_DraftBlueprint_v01\geographic_summaries\PBA50Plus_DraftBlueprint_v01_county_summary_2025.csv => no diffs (baus-env-2023) E:\LUModel2Share\GitHub\bayarea_urbansim>git diff M:\urban_modeling\baus\PBA50Plus\PBA50Plus_DraftBlueprint\PBA50Plus_Draft_Blueprint_v8_znupd_nodevfix\geographic_summaries\PBA50Plus_Draft_Blueprint_v8_znupd_nodevfix_county_summary_2050.csv E:\LUModel2Share\PBA50Plus_DraftBlueprint\PBA50Plus_DraftBlueprint_v01\geographic_summaries\PBA50Plus_DraftBlueprint_v01_county_summary_2050.csv => no diffs

PBA50Plus_DraftBlueprint_v02 results are identical to PBA50+ DBP

…rder

(baus-env-2023) E:\LUModel2Share\GitHub\bayarea_urbansim>git diff M:\urban_modeling\baus\PBA50Plus\PBA50Plus_DraftBlueprint\PBA50Plus_Draft_Blueprint_v8_znupd_nodevfix\geographic_summaries\PBA50Plus_Draft_Blueprint_v8_znupd_nodevfix_county_summary_2050.csv E:\LUModel2Share\PBA50Plus_DraftBlueprint\PBA50Plus_DraftBlueprint_v03\geographic_summaries\county_summary_2050.csv [no diffs] (baus-env-2023) E:\LUModel2Share\GitHub\bayarea_urbansim>git diff M:\urban_modeling\baus\PBA50Plus\PBA50Plus_DraftBlueprint\PBA50Plus_Draft_Blueprint_v8_znupd_nodevfix\travel_model_summaries\PBA50Plus_Draft_Blueprint_v8_znupd_nodevfix_taz1_summary_2050.csv E:\LUModel2Share\PBA50Plus_DraftBlueprint\PBA50Plus_DraftBlueprint_v03\travel_model_summaries\taz1_summary_2050.csv [no diffs[

And log about files read

And outputs the 2020 h5 file

Also add urbansim logging (but turn it off)

Since it's not really a summary Also add disaggregate households, jobs, and static parcels

parcel_id is the standard parcel identifier

As well as vacant_job_spaces and jobs index

…vels

…coffer

Also print() -> logger.debug()

Prefix print statements for later conversion to log statements

And minimize diffs

akselx

Looks like a major improvement. I assume this runs from snout to tail - I like the many efficiencies and cleanings - will need to adjust off-model scripts to handle the simplified naming convensions, but seems worthwhile.

akselx · 2024-11-22T06:12:44Z

baus/datasources.py

@@ -28,8 +28,10 @@
 # this is similar to the code for settings in urbansim_defaults
 @orca.injectable('run_setup', cache=True)
 def run_setup():
-    with open("run_setup.yaml") as f:
-        return yaml.load(f)
+    run_setup_yaml = orca.get_injectable('run_setup_yaml')


This is a very useful addition

akselx · 2024-11-22T06:17:48Z

baus/datasources.py

+def developer_settings(base_year):
+    developer_settings_yaml = "developer_settings.yaml"
+    full_path = os.path.join(misc.configs_dir(), "developer", developer_settings_yaml)
+    logger.info("Reading developer settings from {}".format(full_path))


I like this addition - logging the files loaded at run time. It is not always clear from the setup file and the setup file is more likely to get corrupted than the log file. I may do that for all the yaml settings loaded.

akselx · 2024-11-22T06:42:50Z

baus/models.py

+    static_parcels_list = developer_settings["static_parcels"]
+    print("static_parcels(): {}".format(static_parcels_list))
+
+    return static_parcels_list


Nice simplification - and I see you got the corresponding source file, as appropriate too for consistency.

akselx · 2024-11-22T06:44:01Z

baus/models.py

        rates = rates.reset_index(drop=True)

-    df = pd.merge(households.to_frame(["zone_id", "base_income_quartile", "tenure"]), rates, on=["zone_id", "base_income_quartile", "tenure"], how="left")
+    df = pd.merge(households.to_frame(["zone_id", "base_income_quartile", "tenure", "move_in_year"]), 


Love the move_in_year addition

akselx · 2024-11-22T07:15:24Z

baus/models.py


    l1 = len(buildings)
    # the following function has `demolish` as an input, but it is not removing the buildings in the 'demolish' table,
    # instead, it removes existing buildings on parcels to be occupied by buildings in 'demolish'   
    buildings = utils._remove_developed_buildings(buildings.to_frame(buildings.local_columns), demolish,
                                                  unplace_agents=["households", "jobs"])
-    orca.add_injectable('static_parcels', np.append(static_parcels, demolish.loc[demolish.action == 'build', 'parcel_id']))
+    # TODO: I don't know if this line makes sense; why would this go on static_parcels?


Ya, that ended up being a long list - instead of just handling on the developer model config side where you can set filters.

akselx · 2024-11-22T22:17:24Z

baus/debug.py

This looks useful. Do you have downstream steps for more or less systematically inspecting the outputs? I could imagine a bunch of vizzes built on top of these.

akselx · 2024-11-22T22:20:49Z

baus.py

@@ -162,311 +170,281 @@ def run_models(mode, run_setup, years_to_run):

    elif mode == "simulation":


We should really move all the run lists to a separate module, or config file, out of baus.py

akselx · 2024-11-22T22:23:06Z

baus/subsidies.py

@@ -141,7 +141,7 @@ def acct_settings(account_strategies):


 @orca.step()
-def lump_sum_accounts(year, years_per_iter, run_setup):
+def lump_sum_accounts(year, base_year, years_per_iter, run_setup, logger):


Convention: when do we pass logger as an arg as opposed to invoking at the top, like getlogger?

akselx · 2024-11-22T22:27:34Z

baus/variables.py

@@ -26,7 +26,7 @@
 #####################


-@orca.column('households', cache=True)
+@orca.column('households', cache=True, cache_scope='step')


So this is the secret - much smaller caching scope.

akselx · 2024-11-22T22:27:55Z

configs/developer/developer_settings.yaml

 # which ignore this list
 static_parcels:
-  - 2489683204304 # ala fairgrounds 378337


lmz added 30 commits July 9, 2024 10:14

Move slack to simplistic submodel for more information on simulation

a66c4b6

Fix copy/paste error

a1eec4b

Make run setup yaml an argument so multiple versions can run at once …

4dc4060

…with the same code

Fix copy paste/error correctly this time

1f636ba

Fix slack error call

5dc485c

Commit file used for DraftBlueprint run

35ac32f

From M:\urban_modeling\baus\PBA50Plus\PBA50Plus_DraftBlueprint\PBA50Plus_Draft_Blueprint_v8_znupd_nodevfix\run_setup_PBA50Plus_Draft_Blueprint_v8_znupd_nodevfix.yaml

Making base year/final year configurable via yaml

5832b11

This commit reproduces PBA50+ DBP run

One more slack fix

98d3ee4

Remove pointless local methods which just return arrays.

dbf847a

PBA50Plus_DraftBlueprint_v02 results are identical to PBA50+ DBP

Stop prefixing every output file with the run name

be85fb6

Sort column names for debug print since they come up in random(ish) o…

9046265

…rder

Testing end in 2030

3246cc3

Differentiate between final_year and stop_year

743f966

Use base_year instead of 2010

5c15e6b

Update to create a 2020-based store

a025ed2

Use yaml.safe_load() to prevent warnings

8ecb332

And log about files read

For 2020-based run, don't run special baseyear models

9a00514

Fix bugs so the h5 table is successfully written

21b856a

This version produces results consistent with PBA50+ DBP

0931807

And outputs the 2020 h5 file

Add residential_units and preproc suffix

7ac2193

Don't create the 2020 file for the start-in-2020 year run

0b7464e

Use the 2020 year base store for the 2020-year start run

63ae3f6

Move debug to the start of the simulation models

e7b6b06

Also add urbansim logging (but turn it off)

Rename parcel/building_summary to disaggregate_output

53b53c1

Since it's not really a summary Also add disaggregate households, jobs, and static parcels

Specify static parcels using parcel_id, not geom_id

de5176e

parcel_id is the standard parcel identifier

Fix for rename of disaggregate output files

979d399

Add move_in_year for households and jobs

5a8cada

Add residential units to disaggregate output

06b28b8

As well as vacant_job_spaces and jobs index

lmz added 28 commits October 17, 2024 12:28

Merge branch 'main' into configurable_base_year

f761618

Fix merge error

23aac59

Logging: write new file (don't append) and turn down some other logle…

174d281

…vels

Fixing more merge errors in baus.py with slack/asana

a2d9abb

Put the baus logger into orca

8de34fc

Merge branch 'main' into configurable_base_year

5cd76dc

Remove unnecessary line given Aksel's last commit

b56147f

Read developer_settings_2020.yaml for 2020 version of static parcels

858b0b4

Convert print to logger.debug and log acct_settings, limit_settings, …

3c5879f

…coffer

Follow-up commit to dropping run_name from output files

1eae966

Fixes to make 2020 baseyear work and reproduce the same results

9a02290

Also print() -> logger.debug()

Back off caching as these can change

605ed6c

Turn down logging for other files

b46f614

Prefix print statements for later conversion to log statements

Increment run_name

6334251

Update 2020 input

04c153a

Doing 2050 test

51d8125

Fix debug logging

a175a07

Add special case for baseyear 2020 to match 2010

7bbd24b

Use options.run_setup_yaml rather than 'run_setup.yaml'

ff26a92

Don't write 2020 data store every time

709ba04

Continue using DBP yaml for rerun

4a623ff

Test removing this

b81495b

Comment updates to DBP yaml

96f439f

Remove test file

fd99e07

Rename for clarity

4dbf0e8

And minimize diffs

Update viz_dir (after the runs)

ddcd753

Deprecated; this isn't used

50ee416

Fix comment

c5888da

lmz requested a review from akselx November 9, 2024 22:32

akselx approved these changes Nov 22, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create configurable base year - so we can start from 2010 or 2020 and achieve the same result #384

Create configurable base year - so we can start from 2010 or 2020 and achieve the same result #384

lmz commented Oct 17, 2024 •

edited

Loading

akselx left a comment

akselx Nov 22, 2024

akselx Nov 22, 2024

akselx Nov 22, 2024

akselx Nov 22, 2024

akselx Nov 22, 2024

akselx Nov 22, 2024

akselx Nov 22, 2024

akselx Nov 22, 2024

akselx Nov 22, 2024

akselx Nov 22, 2024

		@@ -162,311 +170,281 @@ def run_models(mode, run_setup, years_to_run):

		elif mode == "simulation":

Create configurable base year - so we can start from 2010 or 2020 and achieve the same result #384

Are you sure you want to change the base?

Create configurable base year - so we can start from 2010 or 2020 and achieve the same result #384

Conversation

lmz commented Oct 17, 2024 • edited Loading

akselx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lmz commented Oct 17, 2024 •

edited

Loading