From 923c6868a8b4b05c7d097cb8cf60c370aa72ca98 Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Fri, 4 Mar 2022 12:40:46 -0800 Subject: [PATCH 01/28] Add support to specify global build defines and options A script manages the use of a file with a unique name, like `SketchName.ino.globals.h`, in the Sketch source directory to provide compiler command-line options (build options) and sketch global defines. The build option data is encapsulated in a unique "C" comment block and extracted into the build tree during prebuild. --- doc/faq/a06-global-build-options.rst | 129 +++++++++ doc/faq/readme.rst | 9 + platform.txt | 16 +- tools/mkbuildoptglobals.py | 394 +++++++++++++++++++++++++++ 4 files changed, 546 insertions(+), 2 deletions(-) create mode 100644 doc/faq/a06-global-build-options.rst create mode 100644 tools/mkbuildoptglobals.py diff --git a/doc/faq/a06-global-build-options.rst b/doc/faq/a06-global-build-options.rst new file mode 100644 index 0000000000..ba8b26db78 --- /dev/null +++ b/doc/faq/a06-global-build-options.rst @@ -0,0 +1,129 @@ +How to specify global build defines and options +=============================================== + +To create global defines for a Sketch, create a file with a name based + on your sketch’s file name followed by ``.globals.h`` in the Sketch folder. + For example, if the main Sketch file is named +``LowWatermark.ino``, its global defines file would be +``LowWatermark.ino.globals.h``. This file will be implicitly included +with every module built for your Sketch. Do not directly include it in +any of your sketch files or in any other source files. There is no need +to create empty/dummy files, when not used. + +This global define ``.h`` also supports embedding compiler command-line +options in a unique “C” block comment. Compiler options are placed in a +“C” block comment starting with ``/*@create-file:build.opt@``. This +signature line must be alone on a single line. The block comment ending +``*/`` should also be alone on a single line. In between, place your +compiler command-line options just as you would have for the GCC @file +command option. + +Actions taken in processing comment block to create ``build.opt`` \* for +each line, white space is trimmed \* blank lines are skipped \* lines +starting with ``*``, ``//``, or ``#`` are skipped \* the remaining +results are written to build tree\ ``/core/build.opt`` \* ``build.opt`` +is finished with a ``-include ...`` command, which references the global +.h its contents were extracted from. + +Example Sketch: ``LowWatermark.ino`` + +.. code:: cpp + + #include // has prototype for umm_free_heap_size_min() + + void setup() { + Serial.begin(115200); + delay(200); + #ifdef MYTITLE1 + Serial.printf("\r\n" MYTITLE1 MYTITLE2 "\r\n"); + #else + Serial.println("ERROR: MYTITLE1 not present"); + #endif + Serial.printf("Heap Low Watermark %u\r\n", umm_free_heap_size_min()); + } + + void loop() {} + +Global ``.h`` file: ``LowWatermark.ino.globals.h`` + +.. code:: cpp + + /*@create-file:build.opt@ + // An embedded build.opt file using a "C" block comment. The starting signature + // must be on a line by itself. The closing block comment pattern should be on a + // line by itself. Each line within the block comment will be space trimmed and + // written to build.opt, skipping blank lines and lines starting with '//', '*' + // or '#'. + + * this line is ignored + # this line is ignored + -DMYTITLE1="\"Running on \"" + -O3 + //-fanalyzer + -DUMM_STATS_FULL=1 + */ + + #ifndef LOWWATERMARK_INO_GLOBALS_H + #define LOWWATERMARK_INO_GLOBALS_H + + #if !defined(__ASSEMBLER__) + // Defines kept away from assembler modules + // i.e. Defines for .cpp, .ino, .c ... modules + #endif + + #if defined(__cplusplus) + // Defines kept private to .cpp modules + //#pragma message("__cplusplus has been seen") + #define MYTITLE2 "Empty" + #endif + + #if !defined(__cplusplus) && !defined(__ASSEMBLER__) + // Defines kept private to .c modules + #define MYTITLE2 "Full" + #endif + + #if defined(__ASSEMBLER__) + // Defines kept private to assembler modules + #endif + + #endif + +Aggressive Caching of ``core.a`` +================================ + +Using global defines or compiler command-line options will lead to bad +builds when the **Aggressively cache compiled core** feature is enabled. +When ``#define`` changes require ``core.a`` to be recompiled, and +multiple Sketches are open, they can no longer reliably share one cached +``core.a``. In a simple case: The 1st Sketch to be built has its version +of ``core.a`` cached. Other sketches will use this cached version for +their builds. + +To turn this off, you need to find the location of ``preferences.txt``. +Using the Arduino IDE, go to *File->Preferences*. Make note of the path +to ``prefereces.txt``. You cannot edit the file while the Arduino IDE is +running. Close all Arduino IDE windows and edit the file +``preferences.txt``. Change ``compiler.cache_core=true`` to +``compiler.cache_core=false`` and save. Then each sketch will maintain +its *own* copy of ``core.a``. The alternative when using +``compiler.cache_core=true``, is to close all Arduino IDE sketch +windows. Start and run *only* one instance of the IDE, while building a +Sketch that uses global defines. + +Other build confusion +===================== + +1. Renaming files does not change the last modified timestamp, possibly + causing issues when replacing files by renaming and rebuilding. A good + example of this problem would be to have then fixed a typo in file + name ``LowWatermark.ino.globals.h``. You need to touch (update + timestamp) the file so a “rebuild all” is performed. +2. When a ``.h`` file is renamed in the sketch folder, a copy of the old + file remains in the build sketch folder. This can create confusion if + you missed an edit in updating an ``#include`` in one or more of your + modules. That module will continue to use the stale version of the + ``.h`` until you restart the IDE or other major changes that would + cause the IDE to delete and recopy the contents from the source + Sketch directory. Changes on the IDE Tools selection may cause a + complete rebuild, clearing the problem. This may be the culprit for + “What! It built fine last night!” diff --git a/doc/faq/readme.rst b/doc/faq/readme.rst index 53e830358b..40329fea8f 100644 --- a/doc/faq/readme.rst +++ b/doc/faq/readme.rst @@ -191,3 +191,12 @@ How to resolve "undefined reference to ``flashinit`'" error ? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Please read `flash layout <../filesystem.rst>`__ documentation entry. + +How to specify global build defines and options? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Using a unique named `.h` file a `#define` value can be globally accessed. +Additionally, compiler command-line options can be embedded in this file as a +unique block comment. + +`Read more `__. diff --git a/platform.txt b/platform.txt index 8c882ac21e..7d86f7a690 100644 --- a/platform.txt +++ b/platform.txt @@ -17,6 +17,7 @@ runtime.tools.signing={runtime.platform.path}/tools/signing.py runtime.tools.elf2bin={runtime.platform.path}/tools/elf2bin.py runtime.tools.sizes={runtime.platform.path}/tools/sizes.py runtime.tools.makecorever={runtime.platform.path}/tools/makecorever.py +runtime.tools.mkbuildoptglobals={runtime.platform.path}/tools/mkbuildoptglobals.py runtime.tools.mkdir={runtime.platform.path}/tools/mkdir.py runtime.tools.cp={runtime.platform.path}/tools/cp.py runtime.tools.eboot={runtime.platform.path}/bootloaders/eboot/eboot.elf @@ -58,11 +59,18 @@ build.spiffs_start= build.spiffs_end= build.spiffs_blocksize= +# Fully qualified and relative file names for processing sketch global options +globals.h.source.fqfn={build.source.path}/{build.project_name}.globals.h +build.globals.path={build.path}/core +globals.h.fqfn={build.globals.path}/{build.project_name}.globals.h +build.opt.fqfn={build.globals.path}/build.opt +sketchbook.globals.h.rfn= + compiler.path={runtime.tools.xtensa-lx106-elf-gcc.path}/bin/ compiler.sdk.path={runtime.platform.path}/tools/sdk compiler.libc.path={runtime.platform.path}/tools/sdk/libc/xtensa-lx106-elf -compiler.cpreprocessor.flags=-D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -D_GNU_SOURCE -DESP8266 "-I{compiler.sdk.path}/include" "-I{compiler.sdk.path}/{build.lwip_include}" "-I{compiler.libc.path}/include" "-I{build.path}/core" +compiler.cpreprocessor.flags=-D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -D_GNU_SOURCE -DESP8266 @{build.opt.fqfn} "-I{compiler.sdk.path}/include" "-I{compiler.sdk.path}/{build.lwip_include}" "-I{compiler.libc.path}/include" "-I{build.path}/core" # support precompiled libraries in IDE v1.8.6+ compiler.libraries.ldflags= @@ -107,7 +115,11 @@ compiler.elf2hex.extra_flags= ## needs git recipe.hooks.sketch.prebuild.pattern="{runtime.tools.python3.path}/python3" -I "{runtime.tools.signing}" --mode header --publickey "{build.source.path}/public.key" --out "{build.path}/core/Updater_Signing.h" # This is quite a working hack. This form of prebuild hook, while intuitive, is not explicitly documented. -recipe.hooks.prebuild.pattern="{runtime.tools.python3.path}/python3" -I "{runtime.tools.makecorever}" --build_path "{build.path}" --platform_path "{runtime.platform.path}" --version "{version}" +recipe.hooks.prebuild.1.pattern="{runtime.tools.python3.path}/python3" -I "{runtime.tools.makecorever}" --build_path "{build.path}" --platform_path "{runtime.platform.path}" --version "{version}" + +# Handle processing sketch global options +recipe.hooks.prebuild.2.pattern="{runtime.tools.python3.path}/python3" "{runtime.tools.mkbuildoptglobals}" "{globals.h.source.fqfn}" "{globals.h.fqfn}" "{build.opt.fqfn}" "{sketchbook.globals.h.rfn}" + ## Build the app.ld linker file recipe.hooks.linking.prelink.1.pattern="{runtime.tools.python3.path}/python3" -I "{runtime.tools.mkdir}" -p "{build.path}/ld_h/" diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py new file mode 100644 index 0000000000..b82680d685 --- /dev/null +++ b/tools/mkbuildoptglobals.py @@ -0,0 +1,394 @@ +#!/usr/bin/env python3 + +# This script manages the use of a file with a unique name, like +# `SketchName.ino.globals.h`, in the Sketch source directory to provide compiler +# command-line options (build options) and sketch global defines. The build +# option data is encapsulated in a unique "C" comment block and extracted into +# the build tree during prebuild. +# +# Copyright (C) 2022 - M Hightower +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# A Tip of the hat to: +# +# This PR continues the effort to get some form of global build support +# presented by brainelectronics' PR https://github.com/esp8266/Arduino/pull/8095 +# +# Used d-a-v's global name suggestion from arduino PR +# https://github.com/arduino/arduino-cli/pull/1524 +# +""" +Operation + +"SketchName.ino.globals.h" - A global h file in the Source Sketch directory. The +string SketchName is the actual name of the sketch. A matching copy is kept in +the build path/core directory. The file is empty when it does not exist in the +source directory. + +Using SketchName.ino.globals.h as a container to hold build.opt, gives implicit +dependency tracking for build.opt by way of SketchName.ino.globals.h's +dependencies. +Example: + gcc ... @{build.path}/core/build.opt -include "{build.path}/core/{build.project_name}.globals.h" ... + +In this implementation the '-include "{build.path}/core/{build.project_name}.globals.h"' +component is added to the build.opt file. + gcc ... @{build.path}/core/build.opt ... + +At each build cycle, "{build.project_name}.globals.h" is conditoinally copied to +"{build.path}/core/" at prebuild, and build.opt is extraction as needed. The +SketchName.ino.globals.h's dependencies will trigger "rebuild all" as needed. + +If SketchName.ino.globals.h is not in the source sketch folder, an empty +versions is created in the build tree. The file build.opt always contains a +"-include ..." entry so that file dependencies are generated for +SketchName.ino.globals.h. This allows for change detection when the file is +added. +""" + +""" +Arduino `preferences.txt` changes + +"Aggressively cache compiled core" must be turned off for a reliable build process. +In ~/.arduino15/preferences.txt, to disable the feature: + compiler.cache_core=false + +Reference: +https://forum.arduino.cc/t/no-aggressively-cache-compiled-core-in-ide-1-8-15/878954/2 +""" + +""" +# Updates or Additions for platform.txt or platform.local.txt + +runtime.tools.mkbuildoptglobals={runtime.platform.path}/tools/mkbuildoptglobals.py + +# Fully qualified and relative file names for processing sketch global options +globals.h.source.fqfn={build.source.path}/{build.project_name}.globals.h +build.globals.path={build.path}/core +globals.h.fqfn={build.globals.path}/{build.project_name}.globals.h +build.opt.fqfn={build.globals.path}/build.opt +sketchbook.globals.h.rfn= + +recipe.hooks.prebuild.2.pattern="{runtime.tools.python3.path}/python3" "{runtime.tools.mkbuildoptglobals}" "{globals.h.source.fqfn}" "{globals.h.fqfn}" "{build.opt.fqfn}" "{sketchbook.globals.h.rfn}" + +compiler.cpreprocessor.flags=-D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -D_GNU_SOURCE -DESP8266 @{build.opt.path} "-I{compiler.sdk.path}/include" "-I{compiler.sdk.path}/{build.lwip_include}" "-I{compiler.libc.path}/include" "-I{build.path}/core" +""" + +""" +A Sketch.ino.globals.h file with embedded build.opt might look like this + +/*@create-file:build.opt@ +// An embedded build.opt file using a "C" block comment. The starting signature +// must be on a line by itself. The closing block comment pattern should be on a +// line by itself. Each line within the block comment will be space trimmed and +// written to build.opt, skipping blank lines and lines starting with '//', '*' +// or '#'. + +-DMYDEFINE="\"Chimichangas do not exist\"" +-O3 +-fanalyzer +-DUMM_STATS=2 +*/ + +#ifndef SKETCH_INO_GLOBALS_H +#define SKETCH_INO_GLOBALS_H + +#if defined(__cplusplus) +// Defines kept private to .cpp modules +//#pragma message("__cplusplus has been seen") +#endif + +#if !defined(__cplusplus) && !defined(__ASSEMBLER__) +// Defines kept private to .c modules +#endif + +#if defined(__ASSEMBLER__) +// Defines kept private to assembler modules +#endif + +#endif +""" + +""" +Added 2) and 5) to docs + +Caveats, Observations, and Ramblings + +1) Edits to platform.txt or platform.local.txt force a complete rebuild that +removes the core folder. Not a problem, just something to be aware of when +debugging this script. Similarly, changes on the IDE Tools selection cause a +complete rebuild. + +In contrast, the core directory is not deleted when the rebuild occurs from +changing a file with an established dependency. + +2) Renaming files does not change the last modified timestamp, possibly causing +issues when replacing files by renaming and rebuilding. + +A good example of this problem is when you correct the spelling of file +SketchName.ino.globals.h. You need to touch (update time stampt) the file so a +rebuild all is performed. + +3) During the build two identical copies of SketchName.ino.globals.h will exist. +#ifndef fencing will be needed for non comment blocks in SketchName.ino.globals.h. + +4) By using a .h file to encapsulate "build.opt" options, the information is not +lost after a save-as. Before with an individual "build.opt" file, the file was +missing in the saved copy. + +5) When a .h file is renamed, a copy of the old file remains in the build +sketch folder. This can create confusion if you missed an edit in updating an +include in one or more of your modules. That module will continue to use the +stale version of the .h, until you restart the IDE or other major changes that +would cause the IDE to delete and recopy the contents from the source sketch. + +This may be the culprit for "What! It built fine last night!" + +6a) In The case of two Arduino IDE screens up with different programs, they can +share the same core archive file. Defines on one screen will change the core +archive, and a build on the 2nd screen will build with those changes. +The 2nd build will have the core built for the 1st screen. It gets uglier. With +the 2nd program, the newly built modules used headers processed with different +defines than the core. + +6b) Problem: Once core has been build, changes to build.opt or globals.h will +not cause the core archive to be rebuild. You either have to change tool +settings or close and reopen the Arduino IDE. This is a variation on 6a) above. +I thought this was working for the single sketch case, but it does not! :( +That is because sometimes it does build properly. What is unknown are the +causes that will make it work and fail? + * Fresh single Arduino IDE Window, open with file to build - works + +I think these, 6a and 6b, are resolved by setting `compiler.cache_core=false` +in ~/.arduino15/preferences.txt, to disable the aggressive caching feature: + https://forum.arduino.cc/t/no-aggressively-cache-compiled-core-in-ide-1-8-15/878954/2 + +7) Suspected but not confirmed. A quick edit and rebuild don't always work well. +Build does not work as expected. This does not fail often. Maybe PIC NIC. +""" + +import os +import sys +import filecmp +from shutil import copyfile + +def print_msg(*args, **kwargs): + print(*args, flush=True, **kwargs) + + +def print_err(*args, **kwargs): + print(*args, flush=True, file=sys.stderr, **kwargs) # file=sys.stderr, + + +def copy_create_build_file(source_fqfn, build_target_fqfn): + """ + Conditionally copy a newer file between the source directory and the build + directory. When source file is missing, create an empty file in the build + directory. + return True when file change detected. + """ + if os.path.exists(source_fqfn): + if (os.path.exists(build_target_fqfn)) and \ + (os.path.getmtime(build_target_fqfn) >= os.path.getmtime(source_fqfn)): + # only copy newer files - do nothing, all is good + return False + else: + # The new copy gets stamped with the current time, just as other + # files copied by `arduino-builder`. + copyfile(source_fqfn, build_target_fqfn) + else: + if os.path.exists(build_target_fqfn) and \ + os.path.getsize(build_target_fqfn) == 0: + return False + else: + # Place holder - Must have an empty file to satisfy parameter list + # specifications in platform.txt. + open(build_target_fqfn, 'w').close() + return True # file changed + + +def add_include_line(build_opt_fqfn, include_fqfn): + if not os.path.exists(include_fqfn): + # If file is missing, we need an place holder + open(include_fqfn, 'w').close() + build_opt = open(build_opt_fqfn, 'a') + build_opt.write('-include "' + include_fqfn + '"\n') + build_opt.close() + + +def extract_create_build_opt_file(globals_h_fqfn, file_name, build_opt_fqfn): + """ + Extract the embedded build.opt from SketchName.ino.globals.h into build + path/core/build.opt. The subdirectory path must already exist as well as the + copy of SketchName.ino.globals.h. + """ + build_opt = open(build_opt_fqfn, 'w') + if not os.path.exists(globals_h_fqfn) or (0 == os.path.getsize(globals_h_fqfn)): + build_opt.close() + return + # Need to work on signature line used for match to avoid conflicts with + # existing embedded documentation methods. + build_opt_signature = "/*@create-file:build.opt@" + complete_comment = False + build_opt_error = False + line_no = 0 + # If the source sketch did not have the file SketchName.ino.globals.h, an empty + # file was created in the ./core/ folder. + # By using the copy, open will always succeed. + with open(globals_h_fqfn, 'r') as src: + for line in src: + line = line.strip() + line_no += 1 + if line == build_opt_signature: + if complete_comment: + build_opt_error = True + print_err("Multiple embedded build.opt blocks in " + file_name + ":" + str(line_no)) + continue + print_msg("Extracting embedded compiler command-line options from " + file_name + ":" + str(line_no)) + for line in src: + line = line.strip() + line_no += 1 + if 0 == len(line): + continue + if line.startswith("*/"): + complete_comment = True + break + elif line.startswith("*"): # these are so common - skip these should they occur + continue + elif line.startswith("#"): # allow some embedded comments + continue + elif line.startswith("//"): + continue + # some consistency checking before writing - give some hints about what is wrong + elif line == build_opt_signature: + print_err("Double begin before end for embedded build.opt block in " + file_name + ":" + str(line_no)) + build_opt_error = True + elif line.startswith(build_opt_signature): + print_err("build.opt signature block ignored, trailing character for embedded build.opt block in " + file_name + ":" + str(line_no)) + build_opt_error = True + elif "/*" in line or "*/" in line : + print_err("Nesting issue for embedded build.opt block in " + file_name + ":" + str(line_no)) + build_opt_error = True + else: + build_opt.write(line + "\n") + elif line.startswith(build_opt_signature): + print_err("build.opt signature block ignored, trailing character for embedded build.opt block in " + file_name + ":" + str(line_no)) + build_opt_error = True + src.close() + if not complete_comment or build_opt_error: + build_opt.truncate(0) + build_opt.close() + if build_opt_error: + # this will help the script start over when the issue is fixed + os.remove(globals_h_fqfn) + print_err("Extraction failed") + # Don't let the failure get hidden by a spew of nonsensical error + # messages that will follow. Bring things to a halt. + sys.exit(1) + return + elif complete_comment: + print_msg("Created compiler command-line options file " + build_opt_fqfn) + build_opt.close() + + +def get_sketchbook_globals(build_path, sketchbook_globals_path, rebuild_opt_file, build_opt_fqfn): + """ + Construct path to sketchbook globals using relative path from users home directory. + Append to build options only if recomposing build options. + """ + source_fqfn = os.path.expanduser('~/' + sketchbook_globals_path) + notused, file_name = os.path.split(source_fqfn) + build_target_fqfn = os.path.join(build_path, file_name) + copy_create_build_file(source_fqfn, build_target_fqfn) + # The old build.opt will be fine since we did a add_include last time. + if rebuild_opt_file: + add_include_line(build_opt_fqfn, build_target_fqfn) + + +def main(): + if len(sys.argv) >= 4: + source_globals_h_fqfn = sys.argv[1] + globals_name = os.path.basename(source_globals_h_fqfn) + globals_h_fqfn = sys.argv[2] + build_path = os.path.dirname(globals_h_fqfn) + build_opt_fqfn = sys.argv[3] + # Assumption: globals_h_fqfn and build_opt_fqfn have the same dirname + + if len(sys.argv) >= 5: + # Hidden option for more advanced programmers + # Very few things need to be made available globaly to *all* Sketches + # This option can create obfuscation when not used wisely. + # Omit from documentation, assume that only an advanced programmer + # will discover and use this. + sketchbook_globals_path = sys.argv[4] + num_include_lines = 2 + else: + sketchbook_globals_path = "" + num_include_lines = 1 + + if os.path.exists(globals_h_fqfn): + # Check for signs of "Aggressive Caching core.a" + # 1ST time run, build path/core will not exist or be nearly empty, + # nothing can be learned. The presence of globals_h_fqfn in the + # build path/core helps distinguish 1st time run from rebuild. + # This method does not report in all scenarios; however, it does + # report often enough to draw attention to the issue. Some aborted + # builds with incomplete ./core compiles may later produce false + # positives. Only report when globals.h is being used. + if os.path.getsize(globals_h_fqfn) and len(os.listdir(build_path)) < 20: + print_err("Aggressive caching of core.a might be enabled. This may create build errors.") + print_err("Suggest turning off in preferences.txt: \"compiler.cache_core=false\"") + # TODO Revise message with proper URL + print_err("Add URL to topic in docs.") + else: + # Info: When platform.txt, platform.local.txt, or IDE Tools are + # changed, our build path directory was cleaned. Note, + # makecorever.py may have run before us and recreaded the directory. + if not os.path.exists(build_path): + os.makedirs(build_path) + print_msg("Clean build, created dir " + build_path) # dev debug print + + if os.path.exists(source_globals_h_fqfn): + print_msg("Using global defines from " + source_globals_h_fqfn) + + extract_fallback_build_opt = \ + copy_create_build_file(source_globals_h_fqfn, globals_h_fqfn) + + # At this point we have a SketchName.ino.globals.h and build.opt file in + # build path/core/ directory. They may be empty at this stage. + # Reuse old build.opt or extract new one from SketchName.ino.globals.h + if extract_fallback_build_opt: + extract_create_build_opt_file(globals_h_fqfn, globals_name, build_opt_fqfn) + elif os.path.exists(globals_h_fqfn) and os.path.getsize(globals_h_fqfn): + num_lines = sum(1 for line in open(build_opt_fqfn)) + if num_lines > num_include_lines: + print_msg("Using extracted compiler command-line options in " + build_opt_fqfn) + else: + # SketchName.ino.globals.h may have been deleted in the sketch + # directory or is just empty. Recompose build.opt + open(build_opt_fqfn, 'w').close() + extract_fallback_build_opt = True + + if extract_fallback_build_opt: + add_include_line(build_opt_fqfn, globals_h_fqfn) + + if len(sketchbook_globals_path): + get_sketchbook_globals(build_path, sketchbook_globals_path, extract_fallback_build_opt, build_opt_fqfn) + + else: + print_err("Too few arguments. Required arguments:") + print_err(" Source FQFN SketchName.ino.globals.h, Build FQFN SketchName.ino.globals.h, Build FQFN build.opt") + +if __name__ == '__main__': + sys.exit(main()) From ec9bdd8dbff61a51ac5248e720ef77fae194a73f Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Sun, 6 Mar 2022 11:29:52 -0800 Subject: [PATCH 02/28] Applied os.path.normpath() liberally to input arguments. Fixes windows file path issue. Improved helpful message for adding embedded build options. --- tools/mkbuildoptglobals.py | 65 +++++++++++++++++--------------------- 1 file changed, 29 insertions(+), 36 deletions(-) diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index b82680d685..464146def2 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -183,6 +183,11 @@ import filecmp from shutil import copyfile +# Need to work on signature line used for match to avoid conflicts with +# existing embedded documentation methods. +build_opt_signature = "/*@create-file:build.opt@" + + def print_msg(*args, **kwargs): print(*args, flush=True, **kwargs) @@ -233,13 +238,13 @@ def extract_create_build_opt_file(globals_h_fqfn, file_name, build_opt_fqfn): path/core/build.opt. The subdirectory path must already exist as well as the copy of SketchName.ino.globals.h. """ + global build_opt_signature + build_opt = open(build_opt_fqfn, 'w') if not os.path.exists(globals_h_fqfn) or (0 == os.path.getsize(globals_h_fqfn)): build_opt.close() return - # Need to work on signature line used for match to avoid conflicts with - # existing embedded documentation methods. - build_opt_signature = "/*@create-file:build.opt@" + complete_comment = False build_opt_error = False line_no = 0 @@ -296,42 +301,42 @@ def extract_create_build_opt_file(globals_h_fqfn, file_name, build_opt_fqfn): # Don't let the failure get hidden by a spew of nonsensical error # messages that will follow. Bring things to a halt. sys.exit(1) - return + return False elif complete_comment: print_msg("Created compiler command-line options file " + build_opt_fqfn) build_opt.close() + return complete_comment - -def get_sketchbook_globals(build_path, sketchbook_globals_path, rebuild_opt_file, build_opt_fqfn): +def get_sketchbook_globals(build_path, sketchbook_globals_path, build_opt_fqfn): """ Construct path to sketchbook globals using relative path from users home directory. - Append to build options only if recomposing build options. + Append to build options. """ source_fqfn = os.path.expanduser('~/' + sketchbook_globals_path) notused, file_name = os.path.split(source_fqfn) build_target_fqfn = os.path.join(build_path, file_name) copy_create_build_file(source_fqfn, build_target_fqfn) - # The old build.opt will be fine since we did a add_include last time. - if rebuild_opt_file: - add_include_line(build_opt_fqfn, build_target_fqfn) + add_include_line(build_opt_fqfn, build_target_fqfn) def main(): + global build_opt_signature + if len(sys.argv) >= 4: - source_globals_h_fqfn = sys.argv[1] + source_globals_h_fqfn = os.path.normpath(sys.argv[1]) globals_name = os.path.basename(source_globals_h_fqfn) - globals_h_fqfn = sys.argv[2] + globals_h_fqfn = os.path.normpath(sys.argv[2]) build_path = os.path.dirname(globals_h_fqfn) - build_opt_fqfn = sys.argv[3] + build_opt_fqfn = os.path.normpath(sys.argv[3]) # Assumption: globals_h_fqfn and build_opt_fqfn have the same dirname if len(sys.argv) >= 5: - # Hidden option for more advanced programmers + # Hidden option for advanced programmers # Very few things need to be made available globaly to *all* Sketches # This option can create obfuscation when not used wisely. # Omit from documentation, assume that only an advanced programmer # will discover and use this. - sketchbook_globals_path = sys.argv[4] + sketchbook_globals_path = os.path.normpath(sys.argv[4]) num_include_lines = 2 else: sketchbook_globals_path = "" @@ -349,42 +354,30 @@ def main(): if os.path.getsize(globals_h_fqfn) and len(os.listdir(build_path)) < 20: print_err("Aggressive caching of core.a might be enabled. This may create build errors.") print_err("Suggest turning off in preferences.txt: \"compiler.cache_core=false\"") - # TODO Revise message with proper URL - print_err("Add URL to topic in docs.") else: # Info: When platform.txt, platform.local.txt, or IDE Tools are # changed, our build path directory was cleaned. Note, # makecorever.py may have run before us and recreaded the directory. if not os.path.exists(build_path): os.makedirs(build_path) - print_msg("Clean build, created dir " + build_path) # dev debug print + print_msg("Clean build, created dir " + build_path) if os.path.exists(source_globals_h_fqfn): print_msg("Using global defines from " + source_globals_h_fqfn) - extract_fallback_build_opt = \ copy_create_build_file(source_globals_h_fqfn, globals_h_fqfn) - # At this point we have a SketchName.ino.globals.h and build.opt file in - # build path/core/ directory. They may be empty at this stage. - # Reuse old build.opt or extract new one from SketchName.ino.globals.h - if extract_fallback_build_opt: - extract_create_build_opt_file(globals_h_fqfn, globals_name, build_opt_fqfn) - elif os.path.exists(globals_h_fqfn) and os.path.getsize(globals_h_fqfn): - num_lines = sum(1 for line in open(build_opt_fqfn)) - if num_lines > num_include_lines: - print_msg("Using extracted compiler command-line options in " + build_opt_fqfn) - else: - # SketchName.ino.globals.h may have been deleted in the sketch - # directory or is just empty. Recompose build.opt - open(build_opt_fqfn, 'w').close() - extract_fallback_build_opt = True + # globals_h_fqfn timestamp was only updated if the source changed. This + # controls the rebuild on change. We can always extact a new build.opt + # w/o triggering a needless rebuild. + embedded_options = extract_create_build_opt_file(globals_h_fqfn, globals_name, build_opt_fqfn) + if not embedded_options and os.path.exists(source_globals_h_fqfn): + print_msg("To add embedded compiler options, include them in a block comment starting with '" + build_opt_signature + "'." ) - if extract_fallback_build_opt: - add_include_line(build_opt_fqfn, globals_h_fqfn) + add_include_line(build_opt_fqfn, globals_h_fqfn) if len(sketchbook_globals_path): - get_sketchbook_globals(build_path, sketchbook_globals_path, extract_fallback_build_opt, build_opt_fqfn) + get_sketchbook_globals(build_path, sketchbook_globals_path, build_opt_fqfn) else: print_err("Too few arguments. Required arguments:") From dc74e0318fb3dccdb44b6208feb8395507d3d22e Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Sun, 6 Mar 2022 17:14:41 -0800 Subject: [PATCH 03/28] doubleup '\' --- tools/mkbuildoptglobals.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index 464146def2..0991dad6f1 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -228,7 +228,7 @@ def add_include_line(build_opt_fqfn, include_fqfn): # If file is missing, we need an place holder open(include_fqfn, 'w').close() build_opt = open(build_opt_fqfn, 'a') - build_opt.write('-include "' + include_fqfn + '"\n') + build_opt.write('-include "' + include_fqfn.replace('\\', '\\\\') + '"\n') build_opt.close() From 4fdfd3bbfe4b8ae5210b2d3fd4475f9a5c5504d1 Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Mon, 7 Mar 2022 09:33:08 -0800 Subject: [PATCH 04/28] Added context help for build option support --- doc/faq/a06-global-build-options.rst | 24 +++++++++++++----------- tools/mkbuildoptglobals.py | 23 +++++++++++++++++++---- 2 files changed, 32 insertions(+), 15 deletions(-) diff --git a/doc/faq/a06-global-build-options.rst b/doc/faq/a06-global-build-options.rst index ba8b26db78..2ec89bf010 100644 --- a/doc/faq/a06-global-build-options.rst +++ b/doc/faq/a06-global-build-options.rst @@ -2,8 +2,8 @@ How to specify global build defines and options =============================================== To create global defines for a Sketch, create a file with a name based - on your sketch’s file name followed by ``.globals.h`` in the Sketch folder. - For example, if the main Sketch file is named +on your sketch’s file name followed by ``.globals.h`` in the Sketch +folder. For example, if the main Sketch file is named ``LowWatermark.ino``, its global defines file would be ``LowWatermark.ino.globals.h``. This file will be implicitly included with every module built for your Sketch. Do not directly include it in @@ -21,9 +21,10 @@ command option. Actions taken in processing comment block to create ``build.opt`` \* for each line, white space is trimmed \* blank lines are skipped \* lines starting with ``*``, ``//``, or ``#`` are skipped \* the remaining -results are written to build tree\ ``/core/build.opt`` \* ``build.opt`` -is finished with a ``-include ...`` command, which references the global -.h its contents were extracted from. +results are written to build tree\ ``/core/build.opt`` \* multiple +``/*@create-file:build.opt@`` ``*/`` comment blocks are not allowed \* +``build.opt`` is finished with a ``-include ...`` command, which +references the global .h its contents were extracted from. Example Sketch: ``LowWatermark.ino`` @@ -72,7 +73,7 @@ Global ``.h`` file: ``LowWatermark.ino.globals.h`` #endif #if defined(__cplusplus) - // Defines kept private to .cpp modules + // Defines kept private to .cpp and .ino modules //#pragma message("__cplusplus has been seen") #define MYTITLE2 "Empty" #endif @@ -93,14 +94,14 @@ Aggressive Caching of ``core.a`` Using global defines or compiler command-line options will lead to bad builds when the **Aggressively cache compiled core** feature is enabled. -When ``#define`` changes require ``core.a`` to be recompiled, and -multiple Sketches are open, they can no longer reliably share one cached +When ``#define`` changes require rebuilding ``core.a`` and multiple +Sketches are open, they can no longer reliably share one cached ``core.a``. In a simple case: The 1st Sketch to be built has its version of ``core.a`` cached. Other sketches will use this cached version for their builds. To turn this off, you need to find the location of ``preferences.txt``. -Using the Arduino IDE, go to *File->Preferences*. Make note of the path +From the Arduino IDE, go to *File->Preferences*. Make note of the path to ``prefereces.txt``. You cannot edit the file while the Arduino IDE is running. Close all Arduino IDE windows and edit the file ``preferences.txt``. Change ``compiler.cache_core=true`` to @@ -113,11 +114,12 @@ Sketch that uses global defines. Other build confusion ===================== -1. Renaming files does not change the last modified timestamp, possibly - causing issues when replacing files by renaming and rebuilding. A good +1. Renaming a file does not change the last modified timestamp, possibly + causing issues when adding a file by renaming and rebuilding. A good example of this problem would be to have then fixed a typo in file name ``LowWatermark.ino.globals.h``. You need to touch (update timestamp) the file so a “rebuild all” is performed. + 2. When a ``.h`` file is renamed in the sketch folder, a copy of the old file remains in the build sketch folder. This can create confusion if you missed an edit in updating an ``#include`` in one or more of your diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index 0991dad6f1..d1f74e9256 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -187,6 +187,7 @@ # existing embedded documentation methods. build_opt_signature = "/*@create-file:build.opt@" +docs_url = "https://arduino-esp8266.readthedocs.io/en/latest/faq/a06-global-build-options.html" def print_msg(*args, **kwargs): print(*args, flush=True, **kwargs) @@ -307,6 +308,7 @@ def extract_create_build_opt_file(globals_h_fqfn, file_name, build_opt_fqfn): build_opt.close() return complete_comment + def get_sketchbook_globals(build_path, sketchbook_globals_path, build_opt_fqfn): """ Construct path to sketchbook globals using relative path from users home directory. @@ -321,6 +323,7 @@ def get_sketchbook_globals(build_path, sketchbook_globals_path, build_opt_fqfn): def main(): global build_opt_signature + global docs_url if len(sys.argv) >= 4: source_globals_h_fqfn = os.path.normpath(sys.argv[1]) @@ -353,7 +356,9 @@ def main(): # positives. Only report when globals.h is being used. if os.path.getsize(globals_h_fqfn) and len(os.listdir(build_path)) < 20: print_err("Aggressive caching of core.a might be enabled. This may create build errors.") - print_err("Suggest turning off in preferences.txt: \"compiler.cache_core=false\"") + print_err(" Suggest turning off in preferences.txt: \"compiler.cache_core=false\"") + print_err(" Read more at " + docs_url) + else: # Info: When platform.txt, platform.local.txt, or IDE Tools are # changed, our build path directory was cleaned. Note, @@ -371,8 +376,18 @@ def main(): # controls the rebuild on change. We can always extact a new build.opt # w/o triggering a needless rebuild. embedded_options = extract_create_build_opt_file(globals_h_fqfn, globals_name, build_opt_fqfn) - if not embedded_options and os.path.exists(source_globals_h_fqfn): - print_msg("To add embedded compiler options, include them in a block comment starting with '" + build_opt_signature + "'." ) + + # Provide context help for build option support. + source_build_opt_h_fqfn = os.path.join(os.path.dirname(source_globals_h_fqfn), "build_opt.h") + if os.path.exists(source_build_opt_h_fqfn) and not embedded_options: + print_err("Build options file '" + source_build_opt_h_fqfn + "' not supported.") + print_err(" Add build option content to '" + source_globals_h_fqfn + "'.") + print_err(" Embedd compiler command-line options in a block comment starting with '" + build_opt_signature + "'.") + print_err(" Read more at " + docs_url) + elif os.path.exists(source_globals_h_fqfn): + if not embedded_options: + print_msg("Tip: Embedd compiler command-line options in a block comment starting with '" + build_opt_signature + "'.") + print_msg(" Read more at " + docs_url) add_include_line(build_opt_fqfn, globals_h_fqfn) @@ -380,7 +395,7 @@ def main(): get_sketchbook_globals(build_path, sketchbook_globals_path, build_opt_fqfn) else: - print_err("Too few arguments. Required arguments:") + print_err("Too few arguments. Add arguments:") print_err(" Source FQFN SketchName.ino.globals.h, Build FQFN SketchName.ino.globals.h, Build FQFN build.opt") if __name__ == '__main__': From f119feadd7900a2404aec3220d5cd4fb244cf7a3 Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Mon, 7 Mar 2022 14:48:16 -0800 Subject: [PATCH 05/28] expunged sketchbook global added workaround for aggressive caching --- doc/faq/a06-global-build-options.rst | 43 +++++++------- platform.txt | 4 +- tools/mkbuildoptglobals.py | 84 +++++++++++++++++----------- 3 files changed, 78 insertions(+), 53 deletions(-) diff --git a/doc/faq/a06-global-build-options.rst b/doc/faq/a06-global-build-options.rst index 2ec89bf010..7e082e8d56 100644 --- a/doc/faq/a06-global-build-options.rst +++ b/doc/faq/a06-global-build-options.rst @@ -92,24 +92,29 @@ Global ``.h`` file: ``LowWatermark.ino.globals.h`` Aggressive Caching of ``core.a`` ================================ -Using global defines or compiler command-line options will lead to bad -builds when the **Aggressively cache compiled core** feature is enabled. -When ``#define`` changes require rebuilding ``core.a`` and multiple -Sketches are open, they can no longer reliably share one cached -``core.a``. In a simple case: The 1st Sketch to be built has its version -of ``core.a`` cached. Other sketches will use this cached version for -their builds. - -To turn this off, you need to find the location of ``preferences.txt``. -From the Arduino IDE, go to *File->Preferences*. Make note of the path -to ``prefereces.txt``. You cannot edit the file while the Arduino IDE is -running. Close all Arduino IDE windows and edit the file -``preferences.txt``. Change ``compiler.cache_core=true`` to -``compiler.cache_core=false`` and save. Then each sketch will maintain -its *own* copy of ``core.a``. The alternative when using -``compiler.cache_core=true``, is to close all Arduino IDE sketch -windows. Start and run *only* one instance of the IDE, while building a -Sketch that uses global defines. +Without mediation, using global defines or compiler command-line options +could lead to bad builds when the “Aggressively cache compiled core” +feature is enabled. When ``#define`` changes require rebuilding +``core.a`` and multiple Sketches are open, they can no longer reliably +share one cached ``core.a``. In a simple case: The 1st Sketch to be +built has its version of ``core.a`` cached. Other sketches will use this +cached version for their builds. + +When the “Aggressively cache compiled core” feature is enabled and a +global define file is detected, a workaround will turn on and stay on. +When you switch between Sketch windows, core will be recompiled and +cache updated. + +To turn the “Aggressively cache compiled core” feature off, you need to +find the location of ``preferences.txt``. From the Arduino IDE, go to +*File->Preferences*. Make note of the path to ``prefereces.txt``. You +cannot edit the file while the Arduino IDE is running. Close all Arduino +IDE windows and edit the file ``preferences.txt``. Change +``compiler.cache_core=true`` to ``compiler.cache_core=false`` and save. +Then each sketch will maintain its *own* copy of ``core.a``. The +alternative when using ``compiler.cache_core=true``, is to close all +Arduino IDE sketch windows. Start and run *only* one instance of the +IDE, while building a Sketch that uses global defines. Other build confusion ===================== @@ -126,6 +131,6 @@ Other build confusion modules. That module will continue to use the stale version of the ``.h`` until you restart the IDE or other major changes that would cause the IDE to delete and recopy the contents from the source - Sketch directory. Changes on the IDE Tools selection may cause a + Sketch directory. Changes on the IDE Tools board settings may cause a complete rebuild, clearing the problem. This may be the culprit for “What! It built fine last night!” diff --git a/platform.txt b/platform.txt index 7d86f7a690..ebe08cacf8 100644 --- a/platform.txt +++ b/platform.txt @@ -64,7 +64,7 @@ globals.h.source.fqfn={build.source.path}/{build.project_name}.globals.h build.globals.path={build.path}/core globals.h.fqfn={build.globals.path}/{build.project_name}.globals.h build.opt.fqfn={build.globals.path}/build.opt -sketchbook.globals.h.rfn= +commonhfile.fqfn={build.core.path}/CommonHFile.h compiler.path={runtime.tools.xtensa-lx106-elf-gcc.path}/bin/ compiler.sdk.path={runtime.platform.path}/tools/sdk @@ -118,7 +118,7 @@ recipe.hooks.sketch.prebuild.pattern="{runtime.tools.python3.path}/python3" -I " recipe.hooks.prebuild.1.pattern="{runtime.tools.python3.path}/python3" -I "{runtime.tools.makecorever}" --build_path "{build.path}" --platform_path "{runtime.platform.path}" --version "{version}" # Handle processing sketch global options -recipe.hooks.prebuild.2.pattern="{runtime.tools.python3.path}/python3" "{runtime.tools.mkbuildoptglobals}" "{globals.h.source.fqfn}" "{globals.h.fqfn}" "{build.opt.fqfn}" "{sketchbook.globals.h.rfn}" +recipe.hooks.prebuild.2.pattern="{runtime.tools.python3.path}/python3" "{runtime.tools.mkbuildoptglobals}" "{globals.h.source.fqfn}" "{globals.h.fqfn}" "{build.opt.fqfn}" "{commonhfile.fqfn}" ## Build the app.ld linker file diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index d1f74e9256..2c9f40faf6 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -78,9 +78,8 @@ build.globals.path={build.path}/core globals.h.fqfn={build.globals.path}/{build.project_name}.globals.h build.opt.fqfn={build.globals.path}/build.opt -sketchbook.globals.h.rfn= -recipe.hooks.prebuild.2.pattern="{runtime.tools.python3.path}/python3" "{runtime.tools.mkbuildoptglobals}" "{globals.h.source.fqfn}" "{globals.h.fqfn}" "{build.opt.fqfn}" "{sketchbook.globals.h.rfn}" +recipe.hooks.prebuild.2.pattern="{runtime.tools.python3.path}/python3" "{runtime.tools.mkbuildoptglobals}" "{globals.h.source.fqfn}" "{globals.h.fqfn}" "{build.opt.fqfn}" compiler.cpreprocessor.flags=-D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -D_GNU_SOURCE -DESP8266 @{build.opt.path} "-I{compiler.sdk.path}/include" "-I{compiler.sdk.path}/{build.lwip_include}" "-I{compiler.libc.path}/include" "-I{build.path}/core" """ @@ -181,6 +180,7 @@ import os import sys import filecmp +import time from shutil import copyfile # Need to work on signature line used for match to avoid conflicts with @@ -228,6 +228,7 @@ def add_include_line(build_opt_fqfn, include_fqfn): if not os.path.exists(include_fqfn): # If file is missing, we need an place holder open(include_fqfn, 'w').close() + print("add_include_line: Created " + include_fqfn) build_opt = open(build_opt_fqfn, 'a') build_opt.write('-include "' + include_fqfn.replace('\\', '\\\\') + '"\n') build_opt.close() @@ -309,41 +310,49 @@ def extract_create_build_opt_file(globals_h_fqfn, file_name, build_opt_fqfn): return complete_comment -def get_sketchbook_globals(build_path, sketchbook_globals_path, build_opt_fqfn): - """ - Construct path to sketchbook globals using relative path from users home directory. - Append to build options. - """ - source_fqfn = os.path.expanduser('~/' + sketchbook_globals_path) - notused, file_name = os.path.split(source_fqfn) - build_target_fqfn = os.path.join(build_path, file_name) - copy_create_build_file(source_fqfn, build_target_fqfn) - add_include_line(build_opt_fqfn, build_target_fqfn) +def enable_override(enable, commonhfile_fqfn): + file=open(commonhfile_fqfn, 'w') + if enable: + file.write("//Override aggressive caching\n") + file.close() + # enabled when getsize(commonhfile_fqfn) is non-zero, disabled when zero + + +def touch(fname, times=None): + with open(fname, 'a'): + os.utime(fname, times) + + +def time_sync(globals_h_fqfn, commonhfile_fqfn): + ts = time.time() + touch(globals_h_fqfn, (ts, ts)) + touch(commonhfile_fqfn, (ts, ts)) def main(): global build_opt_signature global docs_url + num_include_lines = 1 + use_aggressive_caching_workaround = True + # Enhancement: read preferences.txt and set use_aggressive_caching_workaround + # https://www.arduino.cc/en/hacking/preferences + # :( it can be in 9 different locations - if len(sys.argv) >= 4: + if len(sys.argv) >= 5: source_globals_h_fqfn = os.path.normpath(sys.argv[1]) globals_name = os.path.basename(source_globals_h_fqfn) globals_h_fqfn = os.path.normpath(sys.argv[2]) build_path = os.path.dirname(globals_h_fqfn) build_opt_fqfn = os.path.normpath(sys.argv[3]) # Assumption: globals_h_fqfn and build_opt_fqfn have the same dirname + commonhfile_fqfn = os.path.normpath(sys.argv[4]) - if len(sys.argv) >= 5: - # Hidden option for advanced programmers - # Very few things need to be made available globaly to *all* Sketches - # This option can create obfuscation when not used wisely. - # Omit from documentation, assume that only an advanced programmer - # will discover and use this. - sketchbook_globals_path = os.path.normpath(sys.argv[4]) - num_include_lines = 2 + if os.path.exists(commonhfile_fqfn): + if os.path.getsize(commonhfile_fqfn) and \ + not use_aggressive_caching_workaround: + enable_override(False, commonhfile_fqfn) else: - sketchbook_globals_path = "" - num_include_lines = 1 + enable_override(False, commonhfile_fqfn) if os.path.exists(globals_h_fqfn): # Check for signs of "Aggressive Caching core.a" @@ -354,14 +363,15 @@ def main(): # report often enough to draw attention to the issue. Some aborted # builds with incomplete ./core compiles may later produce false # positives. Only report when globals.h is being used. - if os.path.getsize(globals_h_fqfn) and len(os.listdir(build_path)) < 20: + if not use_aggressive_caching_workaround and \ + os.path.getsize(globals_h_fqfn) and \ + len(os.listdir(build_path)) < 20: print_err("Aggressive caching of core.a might be enabled. This may create build errors.") print_err(" Suggest turning off in preferences.txt: \"compiler.cache_core=false\"") print_err(" Read more at " + docs_url) - else: - # Info: When platform.txt, platform.local.txt, or IDE Tools are - # changed, our build path directory was cleaned. Note, + # Info: When platform.txt, platform.local.txt, or IDE Tools board + # settings are changed, our build path directory was cleaned. Note, # makecorever.py may have run before us and recreaded the directory. if not os.path.exists(build_path): os.makedirs(build_path) @@ -377,6 +387,21 @@ def main(): # w/o triggering a needless rebuild. embedded_options = extract_create_build_opt_file(globals_h_fqfn, globals_name, build_opt_fqfn) + if use_aggressive_caching_workaround: + if os.path.getsize(commonhfile_fqfn): + print("os.path.getmtime(globals_h_fqfn) " + str(os.path.getmtime(globals_h_fqfn))) + print("os.path.getmtime(commonhfile_fqfn) " + str(os.path.getmtime(commonhfile_fqfn))) + if (os.path.getmtime(globals_h_fqfn) != os.path.getmtime(commonhfile_fqfn)): + # Need to rebuild core.a + # touching commonhfile_fqfn in the source core tree will cause rebuild. + time_sync(globals_h_fqfn, commonhfile_fqfn) + elif os.path.getsize(globals_h_fqfn): + enable_override(True, commonhfile_fqfn) + time_sync(globals_h_fqfn, commonhfile_fqfn) + + add_include_line(build_opt_fqfn, commonhfile_fqfn) + add_include_line(build_opt_fqfn, globals_h_fqfn) + # Provide context help for build option support. source_build_opt_h_fqfn = os.path.join(os.path.dirname(source_globals_h_fqfn), "build_opt.h") if os.path.exists(source_build_opt_h_fqfn) and not embedded_options: @@ -389,11 +414,6 @@ def main(): print_msg("Tip: Embedd compiler command-line options in a block comment starting with '" + build_opt_signature + "'.") print_msg(" Read more at " + docs_url) - add_include_line(build_opt_fqfn, globals_h_fqfn) - - if len(sketchbook_globals_path): - get_sketchbook_globals(build_path, sketchbook_globals_path, build_opt_fqfn) - else: print_err("Too few arguments. Add arguments:") print_err(" Source FQFN SketchName.ino.globals.h, Build FQFN SketchName.ino.globals.h, Build FQFN build.opt") From dc3d37cfce6b07fd66713141a919d0dc9646d2b9 Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Tue, 8 Mar 2022 21:37:35 -0800 Subject: [PATCH 06/28] inital pass at searching for and reading preferences.txt --- tools/mkbuildoptglobals.py | 50 +++++++++++++++++++++++++++++++++++--- 1 file changed, 46 insertions(+), 4 deletions(-) diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index 2c9f40faf6..5a86072de9 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -181,6 +181,7 @@ import sys import filecmp import time +import platform from shutil import copyfile # Need to work on signature line used for match to avoid conflicts with @@ -317,6 +318,50 @@ def enable_override(enable, commonhfile_fqfn): file.close() # enabled when getsize(commonhfile_fqfn) is non-zero, disabled when zero +def find_preferences_txt(): + platform_name = platform.system() + # OS Path list from: + # https://www.arduino.cc/en/hacking/preferences + if "Windows" == platform_name: + fqfn = os.path.expanduser("\Arduino15\preferences.txt") # Windows + if os.path.exists(fqfn): + return fqfn + fqfn = os.path.expanduser("\Documents\ArduinoData\preferences.txt") # Windows app version + if os.path.exists(fqfn): + return fqfn + elif "Darwin" == platform_name: + fqfn = os.path.expanduser("~/Library/Arduino15/preferences.txt") # Max OS X + if os.path.exists(fqfn): + return fqfn + elif "Linux" == platform_name: + fqfn = os.path.expanduser("~/.arduino15/preferences.txt") # Linux - works + if os.path.exists(fqfn): + return fqfn + # Where and how would I find this? + # /portable/preferences.txt (when used in portable mode) + return "" + + +def get_preferences_txt(file_fqfn, key): + with open(file_fqfn) as fd: + for line in fd: + name, value = line.partition("=")[::2] + if name.strip().lower() == key: + if value.strip().lower() == 'true': + print_msg("found compiler.cache_core " + value.strip()) #D debug + return True + else: + return False + return True # If we don't find it just assume it is set True + + +def check_preferences_txt(): + file_fqfn = find_preferences_txt() + if file_fqfn == "": + return True # cannot find file assume enabled + print_msg("\nfound preferences " + file_fqfn) #D debug + return get_preferences_txt(file_fqfn, "compiler.cache_core") + def touch(fname, times=None): with open(fname, 'a'): @@ -333,10 +378,7 @@ def main(): global build_opt_signature global docs_url num_include_lines = 1 - use_aggressive_caching_workaround = True - # Enhancement: read preferences.txt and set use_aggressive_caching_workaround - # https://www.arduino.cc/en/hacking/preferences - # :( it can be in 9 different locations + use_aggressive_caching_workaround = check_preferences_txt() #? preliminary if len(sys.argv) >= 5: source_globals_h_fqfn = os.path.normpath(sys.argv[1]) From 8a96eeb6343c81cd2c96f3c3fb6c65bc7bdb3b42 Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Wed, 9 Mar 2022 11:22:06 -0800 Subject: [PATCH 07/28] Correct Windows path for preferences.txt Added portable path for preferences.txt Expanded file timestamp granularity Improved error message printing for Arduino IDE 2.0 RC4 --- tools/mkbuildoptglobals.py | 114 +++++++++++++++++++++++-------------- 1 file changed, 72 insertions(+), 42 deletions(-) diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index 5a86072de9..8c83a5d6f1 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -173,6 +173,9 @@ in ~/.arduino15/preferences.txt, to disable the aggressive caching feature: https://forum.arduino.cc/t/no-aggressively-cache-compiled-core-in-ide-1-8-15/878954/2 +Added workaround for `compiler.cache_core=true` case. +See `if use_aggressive_caching_workaround:` in main(). + 7) Suspected but not confirmed. A quick edit and rebuild don't always work well. Build does not work as expected. This does not fail often. Maybe PIC NIC. """ @@ -191,11 +194,19 @@ docs_url = "https://arduino-esp8266.readthedocs.io/en/latest/faq/a06-global-build-options.html" def print_msg(*args, **kwargs): - print(*args, flush=True, **kwargs) + print(*args, **kwargs) +# I prefer error messages to stand out; however, using stderr for a different +# color does not work on the new Arduino IDE 2.0 RC4. Also, separate pipes, +# buffering, and multiple threads with output can create mixed-up messages. +# Bring attention to errors with a blank line and lines starting with "*** ". +# Let multiple prints buffer to aid them in staying together. def print_err(*args, **kwargs): - print(*args, flush=True, file=sys.stderr, **kwargs) # file=sys.stderr, + if (args[0])[0] != ' ': + print("") + print("*** ", end='') + print(*args, **kwargs) def copy_create_build_file(source_fqfn, build_target_fqfn): @@ -221,18 +232,19 @@ def copy_create_build_file(source_fqfn, build_target_fqfn): else: # Place holder - Must have an empty file to satisfy parameter list # specifications in platform.txt. - open(build_target_fqfn, 'w').close() + with open(build_target_fqfn, 'w'): + pass return True # file changed def add_include_line(build_opt_fqfn, include_fqfn): if not os.path.exists(include_fqfn): # If file is missing, we need an place holder - open(include_fqfn, 'w').close() + with open(include_fqfn, 'w'): + pass print("add_include_line: Created " + include_fqfn) - build_opt = open(build_opt_fqfn, 'a') - build_opt.write('-include "' + include_fqfn.replace('\\', '\\\\') + '"\n') - build_opt.close() + with open(build_opt_fqfn, 'a') as build_opt: + build_opt.write('-include "' + include_fqfn.replace('\\', '\\\\') + '"\n') def extract_create_build_opt_file(globals_h_fqfn, file_name, build_opt_fqfn): @@ -261,7 +273,7 @@ def extract_create_build_opt_file(globals_h_fqfn, file_name, build_opt_fqfn): if line == build_opt_signature: if complete_comment: build_opt_error = True - print_err("Multiple embedded build.opt blocks in " + file_name + ":" + str(line_no)) + print_err(" Multiple embedded build.opt blocks in " + file_name + ":" + str(line_no)) continue print_msg("Extracting embedded compiler command-line options from " + file_name + ":" + str(line_no)) for line in src: @@ -280,27 +292,26 @@ def extract_create_build_opt_file(globals_h_fqfn, file_name, build_opt_fqfn): continue # some consistency checking before writing - give some hints about what is wrong elif line == build_opt_signature: - print_err("Double begin before end for embedded build.opt block in " + file_name + ":" + str(line_no)) + print_err(" Double begin before end for embedded build.opt block in " + file_name + ":" + str(line_no)) build_opt_error = True elif line.startswith(build_opt_signature): - print_err("build.opt signature block ignored, trailing character for embedded build.opt block in " + file_name + ":" + str(line_no)) + print_err(" build.opt signature block ignored, trailing character for embedded build.opt block in " + file_name + ":" + str(line_no)) build_opt_error = True elif "/*" in line or "*/" in line : - print_err("Nesting issue for embedded build.opt block in " + file_name + ":" + str(line_no)) + print_err(" Nesting issue for embedded build.opt block in " + file_name + ":" + str(line_no)) build_opt_error = True else: build_opt.write(line + "\n") elif line.startswith(build_opt_signature): - print_err("build.opt signature block ignored, trailing character for embedded build.opt block in " + file_name + ":" + str(line_no)) + print_err(" build.opt signature block ignored, trailing character for embedded build.opt block in " + file_name + ":" + str(line_no)) build_opt_error = True - src.close() if not complete_comment or build_opt_error: build_opt.truncate(0) build_opt.close() if build_opt_error: # this will help the script start over when the issue is fixed os.remove(globals_h_fqfn) - print_err("Extraction failed") + print_err(" Extraction failed") # Don't let the failure get hidden by a spew of nonsensical error # messages that will follow. Bring things to a halt. sys.exit(1) @@ -312,46 +323,58 @@ def extract_create_build_opt_file(globals_h_fqfn, file_name, build_opt_fqfn): def enable_override(enable, commonhfile_fqfn): - file=open(commonhfile_fqfn, 'w') - if enable: - file.write("//Override aggressive caching\n") - file.close() + with open(commonhfile_fqfn, 'w') as file: + if enable: + file.write("//Override aggressive caching\n") # enabled when getsize(commonhfile_fqfn) is non-zero, disabled when zero + def find_preferences_txt(): platform_name = platform.system() # OS Path list from: # https://www.arduino.cc/en/hacking/preferences - if "Windows" == platform_name: - fqfn = os.path.expanduser("\Arduino15\preferences.txt") # Windows + if "Linux" == platform_name: + # Test for portable 1ST + # /portable/preferences.txt (when used in portable mode) + # For more on portable mode see https://docs.arduino.cc/software/ide-v1/tutorials/PortableIDE + # Working directory must be set to the location of the Arduino IDE executable. + fqfn = "./portable/preferences.txt" # Linux portable - verified if os.path.exists(fqfn): return fqfn - fqfn = os.path.expanduser("\Documents\ArduinoData\preferences.txt") # Windows app version + fqfn = os.path.expanduser("~/.arduino15/preferences.txt") # Linux - verified if os.path.exists(fqfn): return fqfn - elif "Darwin" == platform_name: - fqfn = os.path.expanduser("~/Library/Arduino15/preferences.txt") # Max OS X + elif "Windows" == platform_name: + fqfn = ".\portable\preferences.txt" + if os.path.exists(fqfn): + return fqfn + fqfn = os.path.expanduser("~\Documents\ArduinoData\preferences.txt") # Windows app version - verified if os.path.exists(fqfn): return fqfn - elif "Linux" == platform_name: - fqfn = os.path.expanduser("~/.arduino15/preferences.txt") # Linux - works + fqfn = os.path.expanduser("~\Arduino15\preferences.txt") # Windows if os.path.exists(fqfn): return fqfn - # Where and how would I find this? - # /portable/preferences.txt (when used in portable mode) + elif "Darwin" == platform_name: + # Skip portable on Macs. Portable is not compatable with Macs + # see https://docs.arduino.cc/software/ide-v1/tutorials/PortableIDE + fqfn = os.path.expanduser("~/Library/Arduino15/preferences.txt") # Max OS X + if os.path.exists(fqfn): + return fqfn + + print_err("File preferences.txt not found on " + platform_name) return "" def get_preferences_txt(file_fqfn, key): - with open(file_fqfn) as fd: - for line in fd: + with open(file_fqfn) as file: + for line in file: name, value = line.partition("=")[::2] if name.strip().lower() == key: if value.strip().lower() == 'true': - print_msg("found compiler.cache_core " + value.strip()) #D debug return True else: return False + print_err("Key " + key + " not found in preferences.txt. Default to true.") return True # If we don't find it just assume it is set True @@ -359,7 +382,7 @@ def check_preferences_txt(): file_fqfn = find_preferences_txt() if file_fqfn == "": return True # cannot find file assume enabled - print_msg("\nfound preferences " + file_fqfn) #D debug + print_msg("Using preferences from " + file_fqfn) return get_preferences_txt(file_fqfn, "compiler.cache_core") @@ -368,25 +391,27 @@ def touch(fname, times=None): os.utime(fname, times) -def time_sync(globals_h_fqfn, commonhfile_fqfn): - ts = time.time() - touch(globals_h_fqfn, (ts, ts)) - touch(commonhfile_fqfn, (ts, ts)) +def synchronous_touch(globals_h_fqfn, commonhfile_fqfn): + with open(globals_h_fqfn, 'a'): + os.utime(globals_h_fqfn) + ts = os.stat(globals_h_fqfn) + with open(commonhfile_fqfn, 'a'): + os.utime(commonhfile_fqfn, ns=(ts.st_atime_ns, ts.st_mtime_ns)) def main(): global build_opt_signature global docs_url num_include_lines = 1 - use_aggressive_caching_workaround = check_preferences_txt() #? preliminary + use_aggressive_caching_workaround = check_preferences_txt() if len(sys.argv) >= 5: source_globals_h_fqfn = os.path.normpath(sys.argv[1]) globals_name = os.path.basename(source_globals_h_fqfn) globals_h_fqfn = os.path.normpath(sys.argv[2]) build_path = os.path.dirname(globals_h_fqfn) - build_opt_fqfn = os.path.normpath(sys.argv[3]) # Assumption: globals_h_fqfn and build_opt_fqfn have the same dirname + build_opt_fqfn = os.path.normpath(sys.argv[3]) commonhfile_fqfn = os.path.normpath(sys.argv[4]) if os.path.exists(commonhfile_fqfn): @@ -425,21 +450,26 @@ def main(): copy_create_build_file(source_globals_h_fqfn, globals_h_fqfn) # globals_h_fqfn timestamp was only updated if the source changed. This - # controls the rebuild on change. We can always extact a new build.opt + # controls the rebuild on change. We can always extract a new build.opt # w/o triggering a needless rebuild. embedded_options = extract_create_build_opt_file(globals_h_fqfn, globals_name, build_opt_fqfn) if use_aggressive_caching_workaround: + # When a Sketch owns a "Sketch.ino.globals.h" file in the build tree + # that exactly matches the timestamp of "CommonHFile.h" in the + # platform source tree, it owns the core cache. If not, or + # "Sketch.ino.globals.h" has changed, rebuild core. if os.path.getsize(commonhfile_fqfn): - print("os.path.getmtime(globals_h_fqfn) " + str(os.path.getmtime(globals_h_fqfn))) - print("os.path.getmtime(commonhfile_fqfn) " + str(os.path.getmtime(commonhfile_fqfn))) if (os.path.getmtime(globals_h_fqfn) != os.path.getmtime(commonhfile_fqfn)): # Need to rebuild core.a # touching commonhfile_fqfn in the source core tree will cause rebuild. - time_sync(globals_h_fqfn, commonhfile_fqfn) + # Looks like touching or writing unrelated files in the source core tree will cause rebuild. + synchronous_touch(globals_h_fqfn, commonhfile_fqfn) + print_msg("Using 'aggressive caching' workaround.") elif os.path.getsize(globals_h_fqfn): enable_override(True, commonhfile_fqfn) - time_sync(globals_h_fqfn, commonhfile_fqfn) + synchronous_touch(globals_h_fqfn, commonhfile_fqfn) + print_msg("Using 'aggressive caching' workaround.") add_include_line(build_opt_fqfn, commonhfile_fqfn) add_include_line(build_opt_fqfn, globals_h_fqfn) From 1e0bf97df646829c3c66fb354fc2a52df9b1a1f2 Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Thu, 10 Mar 2022 08:33:42 -0800 Subject: [PATCH 08/28] Improved portable path and various Windows paths to preferences.txt --- doc/faq/a06-global-build-options.rst | 29 +++++---- platform.txt | 2 +- tools/mkbuildoptglobals.py | 89 +++++++++++++++++----------- 3 files changed, 73 insertions(+), 47 deletions(-) diff --git a/doc/faq/a06-global-build-options.rst b/doc/faq/a06-global-build-options.rst index 7e082e8d56..710cb69106 100644 --- a/doc/faq/a06-global-build-options.rst +++ b/doc/faq/a06-global-build-options.rst @@ -92,6 +92,13 @@ Global ``.h`` file: ``LowWatermark.ino.globals.h`` Aggressive Caching of ``core.a`` ================================ +The feature “Aggressive Caching of core.a” refers to sharing a single +copy of ``core.a`` across all Arduino IDE Sketch windows. This feature +is on by default. ``core.a`` is an archive file containing the compiled +objects of ``./core/esp8266/*``. Created after your 1ST successful +compilation. All other open sketch builds use this shared file. When you +close all Arduino IDE windows, the core archive file is deleted. + Without mediation, using global defines or compiler command-line options could lead to bad builds when the “Aggressively cache compiled core” feature is enabled. When ``#define`` changes require rebuilding @@ -102,19 +109,19 @@ cached version for their builds. When the “Aggressively cache compiled core” feature is enabled and a global define file is detected, a workaround will turn on and stay on. -When you switch between Sketch windows, core will be recompiled and +When you switch between Sketch windows, core will be recompiled and the cache updated. -To turn the “Aggressively cache compiled core” feature off, you need to -find the location of ``preferences.txt``. From the Arduino IDE, go to -*File->Preferences*. Make note of the path to ``prefereces.txt``. You -cannot edit the file while the Arduino IDE is running. Close all Arduino -IDE windows and edit the file ``preferences.txt``. Change -``compiler.cache_core=true`` to ``compiler.cache_core=false`` and save. -Then each sketch will maintain its *own* copy of ``core.a``. The -alternative when using ``compiler.cache_core=true``, is to close all -Arduino IDE sketch windows. Start and run *only* one instance of the -IDE, while building a Sketch that uses global defines. +If you think your workflow performance would benefit from keeping a per +Sketch copy of ``core.a``, you can turn off the “Aggressively cache +compiled core” feature. You need to find the location of +``preferences.txt``. From the Arduino IDE, go to *File->Preferences*. +Make note of the path to ``prefereces.txt``. You cannot edit the file +while the Arduino IDE is running. Close all Arduino IDE windows and edit +the file ``preferences.txt``. Change ``compiler.cache_core=true`` to +``compiler.cache_core=false`` and save. Then each sketch will maintain +its *own* copy of ``core.a`` built with the customization expressed by +their respective ``build.opt`` file. Other build confusion ===================== diff --git a/platform.txt b/platform.txt index ebe08cacf8..c92c3b30b2 100644 --- a/platform.txt +++ b/platform.txt @@ -118,7 +118,7 @@ recipe.hooks.sketch.prebuild.pattern="{runtime.tools.python3.path}/python3" -I " recipe.hooks.prebuild.1.pattern="{runtime.tools.python3.path}/python3" -I "{runtime.tools.makecorever}" --build_path "{build.path}" --platform_path "{runtime.platform.path}" --version "{version}" # Handle processing sketch global options -recipe.hooks.prebuild.2.pattern="{runtime.tools.python3.path}/python3" "{runtime.tools.mkbuildoptglobals}" "{globals.h.source.fqfn}" "{globals.h.fqfn}" "{build.opt.fqfn}" "{commonhfile.fqfn}" +recipe.hooks.prebuild.2.pattern="{runtime.tools.python3.path}/python3" "{runtime.tools.mkbuildoptglobals}" "{globals.h.source.fqfn}" "{globals.h.fqfn}" "{build.opt.fqfn}" "{commonhfile.fqfn}" "{runtime.ide.path}" ## Build the app.ld linker file diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index 8c83a5d6f1..019de28e5e 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # This script manages the use of a file with a unique name, like -# `SketchName.ino.globals.h`, in the Sketch source directory to provide compiler +# `Sketch.ino.globals.h`, in the Sketch source directory to provide compiler # command-line options (build options) and sketch global defines. The build # option data is encapsulated in a unique "C" comment block and extracted into # the build tree during prebuild. @@ -31,13 +31,13 @@ """ Operation -"SketchName.ino.globals.h" - A global h file in the Source Sketch directory. The -string SketchName is the actual name of the sketch. A matching copy is kept in -the build path/core directory. The file is empty when it does not exist in the -source directory. +"Sketch.ino.globals.h" - A global h file in the Source Sketch directory. The +string Sketch.ino is the actual name of the sketch program. A matching copy is +kept in the build path/core directory. The file is empty when it does not exist +in the source directory. -Using SketchName.ino.globals.h as a container to hold build.opt, gives implicit -dependency tracking for build.opt by way of SketchName.ino.globals.h's +Using Sketch.ino.globals.h as a container to hold build.opt, gives implicit +dependency tracking for build.opt by way of Sketch.ino.globals.h's dependencies. Example: gcc ... @{build.path}/core/build.opt -include "{build.path}/core/{build.project_name}.globals.h" ... @@ -48,12 +48,12 @@ At each build cycle, "{build.project_name}.globals.h" is conditoinally copied to "{build.path}/core/" at prebuild, and build.opt is extraction as needed. The -SketchName.ino.globals.h's dependencies will trigger "rebuild all" as needed. +Sketch.ino.globals.h's dependencies will trigger "rebuild all" as needed. -If SketchName.ino.globals.h is not in the source sketch folder, an empty +If Sketch.ino.globals.h is not in the source sketch folder, an empty versions is created in the build tree. The file build.opt always contains a "-include ..." entry so that file dependencies are generated for -SketchName.ino.globals.h. This allows for change detection when the file is +Sketch.ino.globals.h. This allows for change detection when the file is added. """ @@ -136,11 +136,11 @@ issues when replacing files by renaming and rebuilding. A good example of this problem is when you correct the spelling of file -SketchName.ino.globals.h. You need to touch (update time stampt) the file so a +Sketch.ino.globals.h. You need to touch (update time stampt) the file so a rebuild all is performed. -3) During the build two identical copies of SketchName.ino.globals.h will exist. -#ifndef fencing will be needed for non comment blocks in SketchName.ino.globals.h. +3) During the build two identical copies of Sketch.ino.globals.h will exist. +#ifndef fencing will be needed for non comment blocks in Sketch.ino.globals.h. 4) By using a .h file to encapsulate "build.opt" options, the information is not lost after a save-as. Before with an individual "build.opt" file, the file was @@ -249,9 +249,9 @@ def add_include_line(build_opt_fqfn, include_fqfn): def extract_create_build_opt_file(globals_h_fqfn, file_name, build_opt_fqfn): """ - Extract the embedded build.opt from SketchName.ino.globals.h into build + Extract the embedded build.opt from Sketch.ino.globals.h into build path/core/build.opt. The subdirectory path must already exist as well as the - copy of SketchName.ino.globals.h. + copy of Sketch.ino.globals.h. """ global build_opt_signature @@ -263,7 +263,7 @@ def extract_create_build_opt_file(globals_h_fqfn, file_name, build_opt_fqfn): complete_comment = False build_opt_error = False line_no = 0 - # If the source sketch did not have the file SketchName.ino.globals.h, an empty + # If the source sketch did not have the file Sketch.ino.globals.h, an empty # file was created in the ./core/ folder. # By using the copy, open will always succeed. with open(globals_h_fqfn, 'r') as src: @@ -329,35 +329,53 @@ def enable_override(enable, commonhfile_fqfn): # enabled when getsize(commonhfile_fqfn) is non-zero, disabled when zero -def find_preferences_txt(): +def find_preferences_txt(runtime_ide_path): platform_name = platform.system() - # OS Path list from: - # https://www.arduino.cc/en/hacking/preferences + # OS Path list for Arduino IDE 1.6.0 and newer + # from: https://www.arduino.cc/en/hacking/preferences if "Linux" == platform_name: # Test for portable 1ST # /portable/preferences.txt (when used in portable mode) # For more on portable mode see https://docs.arduino.cc/software/ide-v1/tutorials/PortableIDE - # Working directory must be set to the location of the Arduino IDE executable. - fqfn = "./portable/preferences.txt" # Linux portable - verified + fqfn = os.path.normpath(runtime_ide_path + "/portable/preferences.txt") + # Linux - verified with Arduino IDE 1.8.19 if os.path.exists(fqfn): return fqfn - fqfn = os.path.expanduser("~/.arduino15/preferences.txt") # Linux - verified + fqfn = os.path.expanduser("~/.arduino15/preferences.txt") + # Linux - verified with Arduino IDE 1.8.18 and 2.0 RC5 64bit and AppImage if os.path.exists(fqfn): return fqfn elif "Windows" == platform_name: - fqfn = ".\portable\preferences.txt" + fqfn = os.path.normpath(runtime_ide_path + "\portable\preferences.txt") + # verified on Windows 10 with Arduino IDE 1.8.19 if os.path.exists(fqfn): return fqfn - fqfn = os.path.expanduser("~\Documents\ArduinoData\preferences.txt") # Windows app version - verified + # It is never simple. Arduino from the Windows APP store or the download + # Windows 8 and up option will save "preferences.txt" in one location. + # The downloaded Windows 7 (and up version) will put "preferences.txt" + # in a different location. When both are present due to various possible + # scenarios, use the more modern. + # Note, I am ignoring any permutations you might get into with storing + # and running applications off Network servers. + fqfn = os.path.expanduser("~\Documents\ArduinoData\preferences.txt") + # Path for "Windows app" - verified on Windows 10 with Arduino IDE 1.8.19 + fqfn2 = os.path.expanduser("~\AppData\local\Arduino15\preferences.txt") + # Path for Windows 7 and up - verified on Windows 10 with Arduino IDE 1.8.19 if os.path.exists(fqfn): - return fqfn - fqfn = os.path.expanduser("~\Arduino15\preferences.txt") # Windows - if os.path.exists(fqfn): - return fqfn + if os.path.exists(fqfn2): + print_err("Multiple 'preferences.txt' files found:") + print_err(" " + fqfn) + print_err(" " + fqfn2) + return fqfn + else: + return fqfn + elif os.path.exists(fqfn2): + return fqfn2 elif "Darwin" == platform_name: - # Skip portable on Macs. Portable is not compatable with Macs + # Portable is not compatable with Mac OS X # see https://docs.arduino.cc/software/ide-v1/tutorials/PortableIDE - fqfn = os.path.expanduser("~/Library/Arduino15/preferences.txt") # Max OS X + fqfn = os.path.expanduser("~/Library/Arduino15/preferences.txt") + # Mac OS X - unverified if os.path.exists(fqfn): return fqfn @@ -378,8 +396,8 @@ def get_preferences_txt(file_fqfn, key): return True # If we don't find it just assume it is set True -def check_preferences_txt(): - file_fqfn = find_preferences_txt() +def check_preferences_txt(runtime_ide_path): + file_fqfn = find_preferences_txt(runtime_ide_path) if file_fqfn == "": return True # cannot find file assume enabled print_msg("Using preferences from " + file_fqfn) @@ -403,9 +421,8 @@ def main(): global build_opt_signature global docs_url num_include_lines = 1 - use_aggressive_caching_workaround = check_preferences_txt() - if len(sys.argv) >= 5: + if len(sys.argv) >= 6: source_globals_h_fqfn = os.path.normpath(sys.argv[1]) globals_name = os.path.basename(source_globals_h_fqfn) globals_h_fqfn = os.path.normpath(sys.argv[2]) @@ -413,6 +430,8 @@ def main(): # Assumption: globals_h_fqfn and build_opt_fqfn have the same dirname build_opt_fqfn = os.path.normpath(sys.argv[3]) commonhfile_fqfn = os.path.normpath(sys.argv[4]) + runtime_ide_path = os.path.normpath(sys.argv[5]) + use_aggressive_caching_workaround = check_preferences_txt(runtime_ide_path) if os.path.exists(commonhfile_fqfn): if os.path.getsize(commonhfile_fqfn) and \ @@ -488,7 +507,7 @@ def main(): else: print_err("Too few arguments. Add arguments:") - print_err(" Source FQFN SketchName.ino.globals.h, Build FQFN SketchName.ino.globals.h, Build FQFN build.opt") + print_err(" Source FQFN Sketch.ino.globals.h, Build FQFN Sketch.ino.globals.h, Build FQFN build.opt") if __name__ == '__main__': sys.exit(main()) From 1c1236b3e9a6272e88f66f70f28e25e1aff0ed01 Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Fri, 11 Mar 2022 17:16:55 -0800 Subject: [PATCH 09/28] Add cleanup logic and identify 1st run after IDE restart --- doc/faq/a06-global-build-options.rst | 11 +++- platform.txt | 8 +-- tools/mkbuildoptglobals.py | 95 +++++++++++++++------------- 3 files changed, 65 insertions(+), 49 deletions(-) diff --git a/doc/faq/a06-global-build-options.rst b/doc/faq/a06-global-build-options.rst index 710cb69106..a5b6437765 100644 --- a/doc/faq/a06-global-build-options.rst +++ b/doc/faq/a06-global-build-options.rst @@ -110,7 +110,16 @@ cached version for their builds. When the “Aggressively cache compiled core” feature is enabled and a global define file is detected, a workaround will turn on and stay on. When you switch between Sketch windows, core will be recompiled and the -cache updated. +cache updated. The workaround logic is reset when Arduino IDE is +completely shutdown and restarted. Some operating systems are better at +cleaning up their temp space than others at reboot after a crash. At +least for Windows you may need to manually delete the Arduino temp files +and directories after a crash. Otherwise the workaround logic may be +left on. + +For some Windows systems the temp directory can be found near +``C:\Users\\AppData\Local\Temp\arduino*``. Note ``AppData`` is +a hidden directory. If you think your workflow performance would benefit from keeping a per Sketch copy of ``core.a``, you can turn off the “Aggressively cache diff --git a/platform.txt b/platform.txt index c92c3b30b2..797db4654e 100644 --- a/platform.txt +++ b/platform.txt @@ -59,12 +59,10 @@ build.spiffs_start= build.spiffs_end= build.spiffs_blocksize= -# Fully qualified and relative file names for processing sketch global options +# Fully qualified file names for processing sketch global options globals.h.source.fqfn={build.source.path}/{build.project_name}.globals.h -build.globals.path={build.path}/core -globals.h.fqfn={build.globals.path}/{build.project_name}.globals.h -build.opt.fqfn={build.globals.path}/build.opt commonhfile.fqfn={build.core.path}/CommonHFile.h +build.opt.fqfn={build.path}/core/build.opt compiler.path={runtime.tools.xtensa-lx106-elf-gcc.path}/bin/ compiler.sdk.path={runtime.platform.path}/tools/sdk @@ -118,7 +116,7 @@ recipe.hooks.sketch.prebuild.pattern="{runtime.tools.python3.path}/python3" -I " recipe.hooks.prebuild.1.pattern="{runtime.tools.python3.path}/python3" -I "{runtime.tools.makecorever}" --build_path "{build.path}" --platform_path "{runtime.platform.path}" --version "{version}" # Handle processing sketch global options -recipe.hooks.prebuild.2.pattern="{runtime.tools.python3.path}/python3" "{runtime.tools.mkbuildoptglobals}" "{globals.h.source.fqfn}" "{globals.h.fqfn}" "{build.opt.fqfn}" "{commonhfile.fqfn}" "{runtime.ide.path}" +recipe.hooks.prebuild.2.pattern="{runtime.tools.python3.path}/python3" "{runtime.tools.mkbuildoptglobals}" "{runtime.ide.path}" "{build.path}" "{build.opt.fqfn}" "{globals.h.source.fqfn}" "{commonhfile.fqfn}" ## Build the app.ld linker file diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index 019de28e5e..6edb08af30 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -73,13 +73,12 @@ runtime.tools.mkbuildoptglobals={runtime.platform.path}/tools/mkbuildoptglobals.py -# Fully qualified and relative file names for processing sketch global options +# Fully qualified file names for processing sketch global options globals.h.source.fqfn={build.source.path}/{build.project_name}.globals.h -build.globals.path={build.path}/core -globals.h.fqfn={build.globals.path}/{build.project_name}.globals.h -build.opt.fqfn={build.globals.path}/build.opt +commonhfile.fqfn={build.core.path}/CommonHFile.h +build.opt.fqfn={build.path}/core/build.opt -recipe.hooks.prebuild.2.pattern="{runtime.tools.python3.path}/python3" "{runtime.tools.mkbuildoptglobals}" "{globals.h.source.fqfn}" "{globals.h.fqfn}" "{build.opt.fqfn}" +recipe.hooks.prebuild.2.pattern="{runtime.tools.python3.path}/python3" "{runtime.tools.mkbuildoptglobals}" "{runtime.ide.path}" "{build.path}" "{build.opt.fqfn}" "{globals.h.source.fqfn}" "{commonhfile.fqfn}" compiler.cpreprocessor.flags=-D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -D_GNU_SOURCE -DESP8266 @{build.opt.path} "-I{compiler.sdk.path}/include" "-I{compiler.sdk.path}/{build.lwip_include}" "-I{compiler.libc.path}/include" "-I{build.path}/core" """ @@ -180,12 +179,12 @@ Build does not work as expected. This does not fail often. Maybe PIC NIC. """ +from shutil import copyfile +import glob import os +import platform import sys -import filecmp import time -import platform -from shutil import copyfile # Need to work on signature line used for match to avoid conflicts with # existing embedded documentation methods. @@ -323,10 +322,36 @@ def extract_create_build_opt_file(globals_h_fqfn, file_name, build_opt_fqfn): def enable_override(enable, commonhfile_fqfn): + # Reduce disk IO writes + if os.path.exists(commonhfile_fqfn): + if os.path.getsize(commonhfile_fqfn): # workaround active + if enable: + return + elif not enable: + return with open(commonhfile_fqfn, 'w') as file: if enable: file.write("//Override aggressive caching\n") - # enabled when getsize(commonhfile_fqfn) is non-zero, disabled when zero + # enable workaround when getsize(commonhfile_fqfn) is non-zero, disabled when zero + + +def discover_1st_time_run(build_path): + # Need to know if this is the 1ST compile of the Arduino IDE starting. + # Use empty cache directory as an indicator for 1ST compile. + # Arduino IDE 2.0 RC5 does not cleanup on exist like 1.6.19. Probably for + # debugging like the irregular version number 10607. For RC5 this indicator + # will be true after a reboot instead of a 1ST compile of the IDE starting. + tmp_path, build = os.path.split(build_path) + ide_2_0 = 'arduino-sketch-' + if ide_2_0 == build[:len(ide_2_0)]: + search_path = os.path.join(tmp_path, 'arduino-core-cache/*') # Arduino IDE 2.0 + else: + search_path = os.path.join(tmp_path, 'arduino_cache_*/*') # Arduino IDE 1.6.x and up + + count = 0 + for dirname in glob.glob(search_path): + count += 1 + return 0 == count def find_preferences_txt(runtime_ide_path): @@ -423,45 +448,27 @@ def main(): num_include_lines = 1 if len(sys.argv) >= 6: - source_globals_h_fqfn = os.path.normpath(sys.argv[1]) - globals_name = os.path.basename(source_globals_h_fqfn) - globals_h_fqfn = os.path.normpath(sys.argv[2]) - build_path = os.path.dirname(globals_h_fqfn) - # Assumption: globals_h_fqfn and build_opt_fqfn have the same dirname + runtime_ide_path = os.path.normpath(sys.argv[1]) + build_path = os.path.normpath(sys.argv[2]) build_opt_fqfn = os.path.normpath(sys.argv[3]) - commonhfile_fqfn = os.path.normpath(sys.argv[4]) - runtime_ide_path = os.path.normpath(sys.argv[5]) + source_globals_h_fqfn = os.path.normpath(sys.argv[4]) + commonhfile_fqfn = os.path.normpath(sys.argv[5]) + + globals_name = os.path.basename(source_globals_h_fqfn) + build_path_core, build_opt_name = os.path.split(build_opt_fqfn) + globals_h_fqfn = os.path.join(build_path_core, globals_name) + + first_time = discover_1st_time_run(build_path) use_aggressive_caching_workaround = check_preferences_txt(runtime_ide_path) - if os.path.exists(commonhfile_fqfn): - if os.path.getsize(commonhfile_fqfn) and \ - not use_aggressive_caching_workaround: - enable_override(False, commonhfile_fqfn) - else: + if first_time or \ + not use_aggressive_caching_workaround or \ + not os.path.exists(commonhfile_fqfn): enable_override(False, commonhfile_fqfn) - if os.path.exists(globals_h_fqfn): - # Check for signs of "Aggressive Caching core.a" - # 1ST time run, build path/core will not exist or be nearly empty, - # nothing can be learned. The presence of globals_h_fqfn in the - # build path/core helps distinguish 1st time run from rebuild. - # This method does not report in all scenarios; however, it does - # report often enough to draw attention to the issue. Some aborted - # builds with incomplete ./core compiles may later produce false - # positives. Only report when globals.h is being used. - if not use_aggressive_caching_workaround and \ - os.path.getsize(globals_h_fqfn) and \ - len(os.listdir(build_path)) < 20: - print_err("Aggressive caching of core.a might be enabled. This may create build errors.") - print_err(" Suggest turning off in preferences.txt: \"compiler.cache_core=false\"") - print_err(" Read more at " + docs_url) - else: - # Info: When platform.txt, platform.local.txt, or IDE Tools board - # settings are changed, our build path directory was cleaned. Note, - # makecorever.py may have run before us and recreaded the directory. - if not os.path.exists(build_path): - os.makedirs(build_path) - print_msg("Clean build, created dir " + build_path) + if not os.path.exists(build_path_core): + os.makedirs(build_path_core) + print_msg("Clean build, created dir " + build_path_core) if os.path.exists(source_globals_h_fqfn): print_msg("Using global defines from " + source_globals_h_fqfn) @@ -478,6 +485,8 @@ def main(): # that exactly matches the timestamp of "CommonHFile.h" in the # platform source tree, it owns the core cache. If not, or # "Sketch.ino.globals.h" has changed, rebuild core. + # A non-zero file size for commonhfile_fqfn, means we have seen a + # globals.h file before and workaround is active. if os.path.getsize(commonhfile_fqfn): if (os.path.getmtime(globals_h_fqfn) != os.path.getmtime(commonhfile_fqfn)): # Need to rebuild core.a From b8e591be68502f9eaccd8999dba26639ee6b4179 Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Sat, 12 Mar 2022 14:26:47 -0800 Subject: [PATCH 10/28] text corrections --- doc/faq/a06-global-build-options.rst | 26 ++++++++++++++------------ doc/faq/readme.rst | 6 +++--- tools/mkbuildoptglobals.py | 22 ++++++++++++---------- 3 files changed, 29 insertions(+), 25 deletions(-) diff --git a/doc/faq/a06-global-build-options.rst b/doc/faq/a06-global-build-options.rst index a5b6437765..af720701fd 100644 --- a/doc/faq/a06-global-build-options.rst +++ b/doc/faq/a06-global-build-options.rst @@ -1,18 +1,18 @@ How to specify global build defines and options =============================================== -To create global defines for a Sketch, create a file with a name based -on your sketch’s file name followed by ``.globals.h`` in the Sketch -folder. For example, if the main Sketch file is named -``LowWatermark.ino``, its global defines file would be +To create globally usable macro definitions for a Sketch, create a file +with a name based on your Sketch’s file name followed by ``.globals.h`` +in the Sketch folder. For example, if the main Sketch file is named +``LowWatermark.ino``, its global ``.h`` file would be ``LowWatermark.ino.globals.h``. This file will be implicitly included with every module built for your Sketch. Do not directly include it in any of your sketch files or in any other source files. There is no need to create empty/dummy files, when not used. -This global define ``.h`` also supports embedding compiler command-line -options in a unique “C” block comment. Compiler options are placed in a -“C” block comment starting with ``/*@create-file:build.opt@``. This +This global ``.h`` also supports embedding compiler command-line options +in a unique “C” block comment. Compiler options are placed in a “C” +block comment starting with ``/*@create-file:build.opt@``. This signature line must be alone on a single line. The block comment ending ``*/`` should also be alone on a single line. In between, place your compiler command-line options just as you would have for the GCC @file @@ -113,13 +113,15 @@ When you switch between Sketch windows, core will be recompiled and the cache updated. The workaround logic is reset when Arduino IDE is completely shutdown and restarted. Some operating systems are better at cleaning up their temp space than others at reboot after a crash. At -least for Windows you may need to manually delete the Arduino temp files -and directories after a crash. Otherwise the workaround logic may be -left on. +least for Windows®, you may need to manually delete the Arduino temp +files and directories after a crash. Otherwise, the workaround logic may +be left on. -For some Windows systems the temp directory can be found near +For some Windows® systems the temp directory can be found near ``C:\Users\\AppData\Local\Temp\arduino*``. Note ``AppData`` is -a hidden directory. +a hidden directory. For help with this do an Internet search on +``windows disk cleanup``. Or, type ``disk cleanup`` in the Windows® +taskbar search box. If you think your workflow performance would benefit from keeping a per Sketch copy of ``core.a``, you can turn off the “Aggressively cache diff --git a/doc/faq/readme.rst b/doc/faq/readme.rst index 40329fea8f..04ccd74880 100644 --- a/doc/faq/readme.rst +++ b/doc/faq/readme.rst @@ -195,8 +195,8 @@ Please read `flash layout <../filesystem.rst>`__ documentation entry. How to specify global build defines and options? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Using a unique named `.h` file a `#define` value can be globally accessed. -Additionally, compiler command-line options can be embedded in this file as a -unique block comment. +By using a uniquely named `.h` file, macro definitions can be created and +globally used. Additionally, compiler command-line options can be embedded in +this file as a unique block comment. `Read more `__. diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index 6edb08af30..35511435f2 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -2,7 +2,7 @@ # This script manages the use of a file with a unique name, like # `Sketch.ino.globals.h`, in the Sketch source directory to provide compiler -# command-line options (build options) and sketch global defines. The build +# command-line options (build options) and sketch global macros. The build # option data is encapsulated in a unique "C" comment block and extracted into # the build tree during prebuild. # @@ -60,7 +60,8 @@ """ Arduino `preferences.txt` changes -"Aggressively cache compiled core" must be turned off for a reliable build process. +"Aggressively cache compiled core" ideally should be turned off; however, +a workaround has been implimented. In ~/.arduino15/preferences.txt, to disable the feature: compiler.cache_core=false @@ -341,6 +342,7 @@ def discover_1st_time_run(build_path): # Arduino IDE 2.0 RC5 does not cleanup on exist like 1.6.19. Probably for # debugging like the irregular version number 10607. For RC5 this indicator # will be true after a reboot instead of a 1ST compile of the IDE starting. + # Another issue for this technique, Windows does not clear the Temp directory. :( tmp_path, build = os.path.split(build_path) ide_2_0 = 'arduino-sketch-' if ide_2_0 == build[:len(ide_2_0)]: @@ -380,10 +382,8 @@ def find_preferences_txt(runtime_ide_path): # The downloaded Windows 7 (and up version) will put "preferences.txt" # in a different location. When both are present due to various possible # scenarios, use the more modern. - # Note, I am ignoring any permutations you might get into with storing - # and running applications off Network servers. fqfn = os.path.expanduser("~\Documents\ArduinoData\preferences.txt") - # Path for "Windows app" - verified on Windows 10 with Arduino IDE 1.8.19 + # Path for "Windows app" - verified on Windows 10 with Arduino IDE 1.8.19 from APP store fqfn2 = os.path.expanduser("~\AppData\local\Arduino15\preferences.txt") # Path for Windows 7 and up - verified on Windows 10 with Arduino IDE 1.8.19 if os.path.exists(fqfn): @@ -422,9 +422,10 @@ def get_preferences_txt(file_fqfn, key): def check_preferences_txt(runtime_ide_path): + # return the state of "compiler.cache_core" in preferences.txt file_fqfn = find_preferences_txt(runtime_ide_path) if file_fqfn == "": - return True # cannot find file assume enabled + return True # cannot find file - assume enabled print_msg("Using preferences from " + file_fqfn) return get_preferences_txt(file_fqfn, "compiler.cache_core") @@ -435,6 +436,7 @@ def touch(fname, times=None): def synchronous_touch(globals_h_fqfn, commonhfile_fqfn): + # touch both files with the same timestamp with open(globals_h_fqfn, 'a'): os.utime(globals_h_fqfn) ts = os.stat(globals_h_fqfn) @@ -481,9 +483,9 @@ def main(): embedded_options = extract_create_build_opt_file(globals_h_fqfn, globals_name, build_opt_fqfn) if use_aggressive_caching_workaround: - # When a Sketch owns a "Sketch.ino.globals.h" file in the build tree - # that exactly matches the timestamp of "CommonHFile.h" in the - # platform source tree, it owns the core cache. If not, or + # When the sketch build has a "Sketch.ino.globals.h" file in the + # build tree that exactly matches the timestamp of "CommonHFile.h" + # in the platform source tree, it owns the core cache. If not, or # "Sketch.ino.globals.h" has changed, rebuild core. # A non-zero file size for commonhfile_fqfn, means we have seen a # globals.h file before and workaround is active. @@ -516,7 +518,7 @@ def main(): else: print_err("Too few arguments. Add arguments:") - print_err(" Source FQFN Sketch.ino.globals.h, Build FQFN Sketch.ino.globals.h, Build FQFN build.opt") + print_err(" Runtime IDE path, Build path, Build FQFN build.opt, Source FQFN Sketch.ino.globals.h, Core Source FQFN CommonHFile.h") if __name__ == '__main__': sys.exit(main()) From 02add35bafedba479c2fcedf1359e8bd10c3c613 Mon Sep 17 00:00:00 2001 From: david gauchard Date: Sun, 13 Mar 2022 23:39:26 +0100 Subject: [PATCH 11/28] Create mkbuildoptglobals.py When global header file does not exist, this print makes it easier for user to create the header file by providing its name and documentation pointer. --- tools/mkbuildoptglobals.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index 35511435f2..4636f5ca83 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -226,6 +226,8 @@ def copy_create_build_file(source_fqfn, build_target_fqfn): # files copied by `arduino-builder`. copyfile(source_fqfn, build_target_fqfn) else: + print_msg("Note: optional global include file '" + source_fqfn + "' does not exist"); + print_msg(" (please check " + docs_url + ")"); if os.path.exists(build_target_fqfn) and \ os.path.getsize(build_target_fqfn) == 0: return False From fc524428424c55e9c9a4335ff5726161134bf34a Mon Sep 17 00:00:00 2001 From: david gauchard Date: Sun, 13 Mar 2022 23:42:53 +0100 Subject: [PATCH 12/28] build.opt heads up to user Compiler command line changes from build.opt are shown to user --- tools/mkbuildoptglobals.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index 4636f5ca83..d48c9b8216 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -303,6 +303,7 @@ def extract_create_build_opt_file(globals_h_fqfn, file_name, build_opt_fqfn): print_err(" Nesting issue for embedded build.opt block in " + file_name + ":" + str(line_no)) build_opt_error = True else: + print_msg("--- additional command-line option: " + line) build_opt.write(line + "\n") elif line.startswith(build_opt_signature): print_err(" build.opt signature block ignored, trailing character for embedded build.opt block in " + file_name + ":" + str(line_no)) From e74857f55a5652805032bcd01fa1977eaf9f3970 Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Sun, 13 Mar 2022 18:23:03 -0700 Subject: [PATCH 13/28] Updated text --- tools/mkbuildoptglobals.py | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index d48c9b8216..14d79c6de2 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -226,8 +226,6 @@ def copy_create_build_file(source_fqfn, build_target_fqfn): # files copied by `arduino-builder`. copyfile(source_fqfn, build_target_fqfn) else: - print_msg("Note: optional global include file '" + source_fqfn + "' does not exist"); - print_msg(" (please check " + docs_url + ")"); if os.path.exists(build_target_fqfn) and \ os.path.getsize(build_target_fqfn) == 0: return False @@ -303,7 +301,7 @@ def extract_create_build_opt_file(globals_h_fqfn, file_name, build_opt_fqfn): print_err(" Nesting issue for embedded build.opt block in " + file_name + ":" + str(line_no)) build_opt_error = True else: - print_msg("--- additional command-line option: " + line) + print_msg(" Add command-line option: " + line) build_opt.write(line + "\n") elif line.startswith(build_opt_signature): print_err(" build.opt signature block ignored, trailing character for embedded build.opt block in " + file_name + ":" + str(line_no)) @@ -320,7 +318,7 @@ def extract_create_build_opt_file(globals_h_fqfn, file_name, build_opt_fqfn): sys.exit(1) return False elif complete_comment: - print_msg("Created compiler command-line options file " + build_opt_fqfn) + print_msg(" Created compiler command-line options file " + build_opt_fqfn) build_opt.close() return complete_comment @@ -476,7 +474,10 @@ def main(): print_msg("Clean build, created dir " + build_path_core) if os.path.exists(source_globals_h_fqfn): - print_msg("Using global defines from " + source_globals_h_fqfn) + print_msg("Using global include from " + source_globals_h_fqfn) + else: + print_msg("Note: optional global include file '" + source_fqfn + "' does not exist"); + print_msg(" (please check " + docs_url + ")"); copy_create_build_file(source_globals_h_fqfn, globals_h_fqfn) From 7932a2421a2408989e4936ea21f5463ef9a59a7c Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Sun, 13 Mar 2022 18:34:20 -0700 Subject: [PATCH 14/28] oops --- tools/mkbuildoptglobals.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index 14d79c6de2..f10358720d 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -476,8 +476,8 @@ def main(): if os.path.exists(source_globals_h_fqfn): print_msg("Using global include from " + source_globals_h_fqfn) else: - print_msg("Note: optional global include file '" + source_fqfn + "' does not exist"); - print_msg(" (please check " + docs_url + ")"); + print_msg("Note: optional global include file '" + source_globals_h_fqfn + "' does not exist.") + print_msg(" (please check " + docs_url + ")") copy_create_build_file(source_globals_h_fqfn, globals_h_fqfn) From 2c45e7134d3d9b058522d1ee958ace62306a7e8f Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Mon, 14 Mar 2022 08:49:55 -0700 Subject: [PATCH 15/28] Expanded comment and made print help consistent --- tools/mkbuildoptglobals.py | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index f10358720d..218b32b5f3 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -477,7 +477,7 @@ def main(): print_msg("Using global include from " + source_globals_h_fqfn) else: print_msg("Note: optional global include file '" + source_globals_h_fqfn + "' does not exist.") - print_msg(" (please check " + docs_url + ")") + print_msg(" Read more at " + docs_url) copy_create_build_file(source_globals_h_fqfn, globals_h_fqfn) @@ -487,10 +487,26 @@ def main(): embedded_options = extract_create_build_opt_file(globals_h_fqfn, globals_name, build_opt_fqfn) if use_aggressive_caching_workaround: + # commonhfile_fqfn encodes the following information + # 1. When touched, it causes a rebuild of core.a + # 2. When file size is non-zero, it indicates we are using the + # aggressive cache workaround. The workaround is set to true + # (active) when we discover a non-zero length global .h file in + # any sketch. The aggressive workaround is cleared on the 1ST + # compile by the Arduino IDE after starting. + # 3. When the timestamp matches the build copy of globals.h + # (globals_h_fqfn), we know one two things: + # * The cached core.a matches up to the current build.opt and + # globals.h. The current sketch owns the cached copy of core.a. + # * globals.h has not changed, and no need to rebuild core.a + # 4. When core.a's timestamp does not match the build copy of + # the global .h file, we only know we need to rebuild core.a, and + # that is enough. + # # When the sketch build has a "Sketch.ino.globals.h" file in the # build tree that exactly matches the timestamp of "CommonHFile.h" - # in the platform source tree, it owns the core cache. If not, or - # "Sketch.ino.globals.h" has changed, rebuild core. + # in the platform source tree, it owns the core.a cache copy. If + # not, or "Sketch.ino.globals.h" has changed, rebuild core. # A non-zero file size for commonhfile_fqfn, means we have seen a # globals.h file before and workaround is active. if os.path.getsize(commonhfile_fqfn): From 9d6c1cb3881891d2b423859b9822a8e8c0b7f42b Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Tue, 15 Mar 2022 14:31:59 -0700 Subject: [PATCH 16/28] Improve handling stderr/stdout with "no verbose output" Grouped helpful info to print at the end. Added missing return value. --- tools/mkbuildoptglobals.py | 69 ++++++++++++++++++++++++++++++-------- 1 file changed, 55 insertions(+), 14 deletions(-) diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index 218b32b5f3..360dc1ee18 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -193,20 +193,58 @@ docs_url = "https://arduino-esp8266.readthedocs.io/en/latest/faq/a06-global-build-options.html" + +err_print_flag = False +msg_print_buf = "" + +# Issues trying to address through buffered printing +# 1. Arduino IDE 2.0 RC5 does not show stderr text in color. Text printed does +# not stand out from stdout messages. +# 2. Separate pipes, buffering, and multiple threads with output can create +# mixed-up messages. "flush" helped but did not resolve. The Arduino IDE 2.0 +# somehow makes the problem worse. +# 3. With Arduino IDE preferences set for "no verbose output", you only see +# stderr messages. Prior related prints are missing. +# +# Locally buffer and merge both stdout and stderr prints. This allows us to +# print a complete context when there is an error. When any buffered prints +# are targeted to stderr, print the whole buffer to stderr. + def print_msg(*args, **kwargs): - print(*args, **kwargs) + global msg_print_buf + # At this time, we can only handle one args value. + msg_print_buf += args[0] + if 'end' in kwargs: + msg_print_buf += kwargs['end'] + else: + msg_print_buf += '\n' -# I prefer error messages to stand out; however, using stderr for a different -# color does not work on the new Arduino IDE 2.0 RC4. Also, separate pipes, -# buffering, and multiple threads with output can create mixed-up messages. # Bring attention to errors with a blank line and lines starting with "*** ". -# Let multiple prints buffer to aid them in staying together. def print_err(*args, **kwargs): + global err_print_flag if (args[0])[0] != ' ': - print("") - print("*** ", end='') - print(*args, **kwargs) + print_msg("") + print_msg("*** ", end='') + print_msg(*args, **kwargs) + err_print_flag = True + + +def handle_error(err_no): + # on err_no 0, commit print buffer to stderr or stdout + # on err_no != 0, commit print buffer to stderr and sys exist with err_no + global msg_print_buf + global err_print_flag + if len(msg_print_buf): + if err_no or err_print_flag: + fd = sys.stderr + else: + fd = sys.stdout + print(msg_print_buf, file=fd, end='', flush=True) + msg_print_buf = "" + err_print_flag = False + if err_no: + sys.exit(err_no) def copy_create_build_file(source_fqfn, build_target_fqfn): @@ -258,7 +296,7 @@ def extract_create_build_opt_file(globals_h_fqfn, file_name, build_opt_fqfn): build_opt = open(build_opt_fqfn, 'w') if not os.path.exists(globals_h_fqfn) or (0 == os.path.getsize(globals_h_fqfn)): build_opt.close() - return + return False complete_comment = False build_opt_error = False @@ -315,8 +353,8 @@ def extract_create_build_opt_file(globals_h_fqfn, file_name, build_opt_fqfn): print_err(" Extraction failed") # Don't let the failure get hidden by a spew of nonsensical error # messages that will follow. Bring things to a halt. - sys.exit(1) - return False + handle_error(1) + return False # not reached elif complete_comment: print_msg(" Created compiler command-line options file " + build_opt_fqfn) build_opt.close() @@ -475,9 +513,6 @@ def main(): if os.path.exists(source_globals_h_fqfn): print_msg("Using global include from " + source_globals_h_fqfn) - else: - print_msg("Note: optional global include file '" + source_globals_h_fqfn + "' does not exist.") - print_msg(" Read more at " + docs_url) copy_create_build_file(source_globals_h_fqfn, globals_h_fqfn) @@ -535,10 +570,16 @@ def main(): if not embedded_options: print_msg("Tip: Embedd compiler command-line options in a block comment starting with '" + build_opt_signature + "'.") print_msg(" Read more at " + docs_url) + else: + print_msg("Note: optional global include file '" + source_globals_h_fqfn + "' does not exist.") + print_msg(" Read more at " + docs_url) else: print_err("Too few arguments. Add arguments:") print_err(" Runtime IDE path, Build path, Build FQFN build.opt, Source FQFN Sketch.ino.globals.h, Core Source FQFN CommonHFile.h") + handle_error(1) + + handle_error(0) # commit print buffer if __name__ == '__main__': sys.exit(main()) From 6b01393a1f935321172f3b1826563c77dec231bb Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Wed, 16 Mar 2022 21:17:59 -0700 Subject: [PATCH 17/28] Correct timestamp on CommonHFile.h More improvements to printing Updated docs. --- doc/faq/a06-global-build-options.rst | 10 +++++++++ tools/mkbuildoptglobals.py | 31 +++++++++++++++++++--------- 2 files changed, 31 insertions(+), 10 deletions(-) diff --git a/doc/faq/a06-global-build-options.rst b/doc/faq/a06-global-build-options.rst index af720701fd..9ffe16f25b 100644 --- a/doc/faq/a06-global-build-options.rst +++ b/doc/faq/a06-global-build-options.rst @@ -123,6 +123,16 @@ a hidden directory. For help with this do an Internet search on ``windows disk cleanup``. Or, type ``disk cleanup`` in the Windows® taskbar search box. +You can run multiple Arduino IDE windows as long as you run one version +of the Arduino IDE at a time. When testing different versions, +completely exit one before starting the next version. For example, +Arduino IDE 1.8.19 and Arduino IDE 2.0 work with different temp and +build paths. With this combination, the workaround logic sometimes fails +to enable. At the time of this writing, when Arduino IDE 2.0 rc5 exits, +it leaves the temp space dirty. This keeps the workaround active the +next time the IDE is started. If this is an issue, manually delete the +temp files. + If you think your workflow performance would benefit from keeping a per Sketch copy of ``core.a``, you can turn off the “Aggressively cache compiled core” feature. You need to find the location of diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index 360dc1ee18..e407514cf2 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -212,8 +212,16 @@ def print_msg(*args, **kwargs): global msg_print_buf - # At this time, we can only handle one args value. + if 'sep' in kwargs: + sep = kwargs['sep'] + else: + sep = ' ' + msg_print_buf += args[0] + for arg in args[1:]: + msg_print_buf += sep + msg_print_buf += arg + if 'end' in kwargs: msg_print_buf += kwargs['end'] else: @@ -225,8 +233,7 @@ def print_err(*args, **kwargs): global err_print_flag if (args[0])[0] != ' ': print_msg("") - print_msg("*** ", end='') - print_msg(*args, **kwargs) + print_msg("***", *args, **kwargs) err_print_flag = True @@ -311,9 +318,9 @@ def extract_create_build_opt_file(globals_h_fqfn, file_name, build_opt_fqfn): if line == build_opt_signature: if complete_comment: build_opt_error = True - print_err(" Multiple embedded build.opt blocks in " + file_name + ":" + str(line_no)) + print_err(" Multiple embedded build.opt blocks in", f'{file_name}:{line_no}') continue - print_msg("Extracting embedded compiler command-line options from " + file_name + ":" + str(line_no)) + print_msg("Extracting embedded compiler command-line options from", f'{file_name}:{line_no}') for line in src: line = line.strip() line_no += 1 @@ -330,19 +337,19 @@ def extract_create_build_opt_file(globals_h_fqfn, file_name, build_opt_fqfn): continue # some consistency checking before writing - give some hints about what is wrong elif line == build_opt_signature: - print_err(" Double begin before end for embedded build.opt block in " + file_name + ":" + str(line_no)) + print_err(" Double begin before end for embedded build.opt block in", f'{file_name}:{line_no}') build_opt_error = True elif line.startswith(build_opt_signature): - print_err(" build.opt signature block ignored, trailing character for embedded build.opt block in " + file_name + ":" + str(line_no)) + print_err(" build.opt signature block ignored, trailing character for embedded build.opt block in", f'{file_name}:{line_no}') build_opt_error = True elif "/*" in line or "*/" in line : - print_err(" Nesting issue for embedded build.opt block in " + file_name + ":" + str(line_no)) + print_err(" Nesting issue for embedded build.opt block in", f'{file_name}:{line_no}') build_opt_error = True else: - print_msg(" Add command-line option: " + line) + print_msg(" ", f'{line_no:2}, Add command-line option: {line}', sep='') build_opt.write(line + "\n") elif line.startswith(build_opt_signature): - print_err(" build.opt signature block ignored, trailing character for embedded build.opt block in " + file_name + ":" + str(line_no)) + print_err(" build.opt signature block ignored, trailing character for embedded build.opt block in", f'{file_name}:{line_no}') build_opt_error = True if not complete_comment or build_opt_error: build_opt.truncate(0) @@ -507,6 +514,10 @@ def main(): not os.path.exists(commonhfile_fqfn): enable_override(False, commonhfile_fqfn) + if time.time_ns() < os.stat(commonhfile_fqfn).st_mtime_ns: + print_err(f"Neutralize future timestamp on build file: {commonhfile_fqfn}") + touch(commonhfile_fqfn) + if not os.path.exists(build_path_core): os.makedirs(build_path_core) print_msg("Clean build, created dir " + build_path_core) From 84dc987be0696a3f5fc5cbbe43c2d216f20a6a2d Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Wed, 23 Mar 2022 10:00:41 -0700 Subject: [PATCH 18/28] Added command-line parser Support hints for compiler.cache_core. For use when Arduino IDE uses command-line options that override compiler.cache_core. Removed overuse of () Improve FAQ entry --- doc/faq/a06-global-build-options.rst | 146 ++++++++++++++++++++------- platform.txt | 3 +- tools/mkbuildoptglobals.py | 141 +++++++++++++++++++++----- 3 files changed, 228 insertions(+), 62 deletions(-) mode change 100644 => 100755 tools/mkbuildoptglobals.py diff --git a/doc/faq/a06-global-build-options.rst b/doc/faq/a06-global-build-options.rst index 9ffe16f25b..0e65e526f7 100644 --- a/doc/faq/a06-global-build-options.rst +++ b/doc/faq/a06-global-build-options.rst @@ -89,33 +89,79 @@ Global ``.h`` file: ``LowWatermark.ino.globals.h`` #endif -Aggressive Caching of ``core.a`` +Aggressively Cache Compiled core ================================ -The feature “Aggressive Caching of core.a” refers to sharing a single -copy of ``core.a`` across all Arduino IDE Sketch windows. This feature -is on by default. ``core.a`` is an archive file containing the compiled -objects of ``./core/esp8266/*``. Created after your 1ST successful -compilation. All other open sketch builds use this shared file. When you -close all Arduino IDE windows, the core archive file is deleted. - -Without mediation, using global defines or compiler command-line options -could lead to bad builds when the “Aggressively cache compiled core” -feature is enabled. When ``#define`` changes require rebuilding -``core.a`` and multiple Sketches are open, they can no longer reliably -share one cached ``core.a``. In a simple case: The 1st Sketch to be -built has its version of ``core.a`` cached. Other sketches will use this -cached version for their builds. - -When the “Aggressively cache compiled core” feature is enabled and a +This feature appeared with the release of Arduino IDE 1.8.2. The feature +“Aggressively Cache Compiled core” refers to sharing a single copy of +``core.a`` across all Arduino IDE Sketch windows. This feature is on by +default. ``core.a`` is an archive file containing the compiled objects +of ``./core/esp8266/*``. Created after your 1ST successful compilation. +All other open sketch builds use this shared file. When you close all +Arduino IDE windows, the core archive file is deleted. + +This feature is not compatible with using global defines or compiler +command-line options. Without mediation, bad builds could result, when +left enabled. When ``#define`` changes require rebuilding ``core.a`` and +multiple Sketches are open, they can no longer reliably share one cached +``core.a``. In a simple case: The 1st Sketch to be built has its version +of ``core.a`` cached. Other sketches will use this cached version for +their builds. + +There are two solutions to this issue: 1. Turn off the “Aggressively +Cache Compiled core” feature, by setting ``compiler.cache_core=false``. +2. Rely on the not ideal fail-safe, aggressive cache workaround built +into the script. + +Using “compiler.cache_core=false” +--------------------------------- + +There are two ways to turn off the “Aggressively Cache Compiled core” +feature: This can be done with the Arduino IDE command-line or a text +editor. + +Using the Arduino IDE command-line from a system command line, enter the +following: + +:: + + arduino --pref compiler.cache_core=false --save-prefs + +For the text editor, you need to find the location of +``preferences.txt``. From the Arduino IDE, go to *File->Preferences*. +Make note of the path to ``prefereces.txt``. You *cannot* edit the file +while the Arduino IDE is running. Close all Arduino IDE windows and edit +the file ``preferences.txt``. Change ``compiler.cache_core=true`` to +``compiler.cache_core=false`` and save. Then each sketch will maintain +its *own* copy of ``core.a`` built with the customization expressed by +their respective ``build.opt`` file. + +The “workaround” +---------------- + +When the “Aggressively Cache Compiled core” feature is enabled and the global define file is detected, a workaround will turn on and stay on. When you switch between Sketch windows, core will be recompiled and the cache updated. The workaround logic is reset when Arduino IDE is -completely shutdown and restarted. Some operating systems are better at -cleaning up their temp space than others at reboot after a crash. At -least for Windows®, you may need to manually delete the Arduino temp -files and directories after a crash. Otherwise, the workaround logic may -be left on. +completely shutdown and restarted. + +The workaround is not perfect. These issues may be of concern: 1. Dirty +temp space. Arduino build cache files left over from a previous run or +boot. 2. Arduino command-line options: \* override default +preferences.txt file. \* override a preference, specifically +``compiler.cache_core``. 3. Multiple versions of the Arduino IDE running + +**Dirty temp space** + +A minor concern, the workaround is always on. Not an issue for build +accuracy, but ``core.a`` maybe rebuild more often than necessary. + +Some operating systems are better at cleaning up their temp space than +others at reboot after a crash. At least for Windows®, you may need to +manually delete the Arduino temp files and directories after a crash. +Otherwise, the workaround logic may be left on. There is no harm in the +workaround being stuck on, the build will be correct; however, the core +files will occasionally be recompiled when not needed. For some Windows® systems the temp directory can be found near ``C:\Users\\AppData\Local\Temp\arduino*``. Note ``AppData`` is @@ -123,26 +169,52 @@ a hidden directory. For help with this do an Internet search on ``windows disk cleanup``. Or, type ``disk cleanup`` in the Windows® taskbar search box. +With Linux, this problem could occur after an Arduino IDE crash. The +problem would be cleared after a reboot. Or you can manually cleanup the +``/tmp/`` directory before restarting the Arduino IDE. + +**Arduino command-line option overrides** + +The script needs to know the working value of ``compiler.cache_core`` +that the Arduino IDE uses when building. This script can learn the state +through documented locations; however, the Arduino IDE has two +command-line options that can alter the results the Arduino IDE uses +internally. And, the Arduino IDE does not provide a means for a script +to learn the override value. + +These two command-line options are the problem: + +:: + + ./arduino --preferences-file other-preferences.txt + ./arduino --pref compiler.cache_core=false + +Hints for discovering the value of ``compiler.cache_core`` use can be +provided by specifying ``mkbuildoptglobals.extra_flags=...`` in +``platform.local.txt``. + +Examples of hints: + +:: + + mkbuildoptglobals.extra_flags=--preferences_sketch # assume file preferences.txt in the sketch folder + mkbuildoptglobals.extra_flags=--preferences_sketch pref.txt # is relative to the sketch folder + mkbuildoptglobals.extra_flags=--no_cache_core + mkbuildoptglobals.extra_flags=--cache_core + mkbuildoptglobals.extra_flags=--preferences_file other-preferences.txt # relative to IDE or full path + +**Multiple versions of the Arduino IDE running** + You can run multiple Arduino IDE windows as long as you run one version of the Arduino IDE at a time. When testing different versions, completely exit one before starting the next version. For example, Arduino IDE 1.8.19 and Arduino IDE 2.0 work with different temp and build paths. With this combination, the workaround logic sometimes fails -to enable. At the time of this writing, when Arduino IDE 2.0 rc5 exits, -it leaves the temp space dirty. This keeps the workaround active the -next time the IDE is started. If this is an issue, manually delete the -temp files. - -If you think your workflow performance would benefit from keeping a per -Sketch copy of ``core.a``, you can turn off the “Aggressively cache -compiled core” feature. You need to find the location of -``preferences.txt``. From the Arduino IDE, go to *File->Preferences*. -Make note of the path to ``prefereces.txt``. You cannot edit the file -while the Arduino IDE is running. Close all Arduino IDE windows and edit -the file ``preferences.txt``. Change ``compiler.cache_core=true`` to -``compiler.cache_core=false`` and save. Then each sketch will maintain -its *own* copy of ``core.a`` built with the customization expressed by -their respective ``build.opt`` file. +to enable. + +At the time of this writing, when Arduino IDE 2.0 rc5 exits, it leaves +the temp space dirty. This keeps the workaround active the next time the +IDE is started. If this is an issue, manually delete the temp files. Other build confusion ===================== diff --git a/platform.txt b/platform.txt index 797db4654e..5295aa62f3 100644 --- a/platform.txt +++ b/platform.txt @@ -63,6 +63,7 @@ build.spiffs_blocksize= globals.h.source.fqfn={build.source.path}/{build.project_name}.globals.h commonhfile.fqfn={build.core.path}/CommonHFile.h build.opt.fqfn={build.path}/core/build.opt +mkbuildoptglobals.extra_flags= compiler.path={runtime.tools.xtensa-lx106-elf-gcc.path}/bin/ compiler.sdk.path={runtime.platform.path}/tools/sdk @@ -116,7 +117,7 @@ recipe.hooks.sketch.prebuild.pattern="{runtime.tools.python3.path}/python3" -I " recipe.hooks.prebuild.1.pattern="{runtime.tools.python3.path}/python3" -I "{runtime.tools.makecorever}" --build_path "{build.path}" --platform_path "{runtime.platform.path}" --version "{version}" # Handle processing sketch global options -recipe.hooks.prebuild.2.pattern="{runtime.tools.python3.path}/python3" "{runtime.tools.mkbuildoptglobals}" "{runtime.ide.path}" "{build.path}" "{build.opt.fqfn}" "{globals.h.source.fqfn}" "{commonhfile.fqfn}" +recipe.hooks.prebuild.2.pattern="{runtime.tools.python3.path}/python3" "{runtime.tools.mkbuildoptglobals}" "{runtime.ide.path}" {runtime.ide.version} "{build.path}" "{build.opt.fqfn}" "{globals.h.source.fqfn}" "{commonhfile.fqfn}" {mkbuildoptglobals.extra_flags} ## Build the app.ld linker file diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py old mode 100644 new mode 100755 index e407514cf2..6e3895b88e --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -180,11 +180,13 @@ Build does not work as expected. This does not fail often. Maybe PIC NIC. """ +import argparse from shutil import copyfile import glob import os import platform import sys +import textwrap import time # Need to work on signature line used for match to avoid conflicts with @@ -262,8 +264,8 @@ def copy_create_build_file(source_fqfn, build_target_fqfn): return True when file change detected. """ if os.path.exists(source_fqfn): - if (os.path.exists(build_target_fqfn)) and \ - (os.path.getmtime(build_target_fqfn) >= os.path.getmtime(source_fqfn)): + if os.path.exists(build_target_fqfn) and \ + os.path.getmtime(build_target_fqfn) >= os.path.getmtime(source_fqfn): # only copy newer files - do nothing, all is good return False else: @@ -459,19 +461,25 @@ def get_preferences_txt(file_fqfn, key): for line in file: name, value = line.partition("=")[::2] if name.strip().lower() == key: - if value.strip().lower() == 'true': - return True - else: - return False - print_err("Key " + key + " not found in preferences.txt. Default to true.") + val = value.strip().lower() + if val != 'true': + val = False + print_msg(f" preferences.txt: {key}={val}") + return val + print_err(" Key " + key + " not found in preferences.txt. Default to true.") return True # If we don't find it just assume it is set True -def check_preferences_txt(runtime_ide_path): - # return the state of "compiler.cache_core" in preferences.txt - file_fqfn = find_preferences_txt(runtime_ide_path) - if file_fqfn == "": - return True # cannot find file - assume enabled +def check_preferences_txt(runtime_ide_path, preferences_file): + # return the state of "compiler.cache_core" found in preferences.txt + file_fqfn = preferences_file + if file_fqfn != None and os.path.exists(file_fqfn): + pass + else: + file_fqfn = find_preferences_txt(runtime_ide_path) + if file_fqfn == "": + return True # cannot find file - assume enabled + print_msg("Using preferences from " + file_fqfn) return get_preferences_txt(file_fqfn, "compiler.cache_core") @@ -489,34 +497,118 @@ def synchronous_touch(globals_h_fqfn, commonhfile_fqfn): with open(commonhfile_fqfn, 'a'): os.utime(commonhfile_fqfn, ns=(ts.st_atime_ns, ts.st_mtime_ns)) +def determine_cache_state(args, runtime_ide_path, source_globals_h_fqfn): + if args.runtime_ide_version < 10802: + return False + elif args.cache_core != None: + print_msg(f"Preferences override, this prebuild script assumes the 'compiler.cache_core' parameter is set to {args.cache_core}") + print_msg(f"To change, modify 'mkbuildoptglobals.extra_flags=(--cache_core | --no_cache_core)' in 'platform.local.txt'") + return args.cache_core + else: + preferences_fqfn = None + if args.preferences_file != None: + preferences_fqfn = args.preferences_file + elif args.preferences_sketch != None: + preferences_fqfn = os.path.normpath( + os.path.join( + os.path.dirname(source_globals_h_fqfn), + args.preferences_sketch)) + elif args.preferences_env != None: + preferences_fqfn = os.getenv(args.preferences_env) + return check_preferences_txt(runtime_ide_path, preferences_fqfn) + + +""" +TODO sort out which of these are viable solutions + +Possible options for handling problems caused by: + ./arduino --preferences-file other-preferences.txt + ./arduino --pref compiler.cache_core=false + +--cache_core +--no_cache_core +--preferences_file (relative to IDE or full path) +--preferences_sketch (default looks for preferences.txt or specify path relative to sketch folder) +--preferences_env, only works on Linux + + export ARDUINO15_PREFERENCES_FILE=$(realpath other-name-than-default-preferences.txt ) + ./arduino --preferences-file other-name-than-default-preferences.txt + + platform.local.txt: mkbuildoptglobals.extra_flags=--preferences_env + + Tested with: + export ARDUINO15_PREFERENCES_FILE=$(realpath ~/projects/arduino/arduino-1.8.19/portable/preferences.txt) + ~/projects/arduino/arduino-1.8.18/arduino + + + Future Issues + * "--preferences-file" does not work for Arduino IDE 2.0, they plan to address at a future release + * Arduino IDE 2.0 does not support portable, they plan to address at a future release + +""" + + +def parse_args(): + extra_txt = '''\ + Use platform.local.txt 'mkbuildoptglobals.extra_flags=...' to supply override options: + --cache_core | --no_cache_core | --preferences_file PREFERENCES_FILE | ... + + more help at {} + '''.format(docs_url) + parser = argparse.ArgumentParser( + description='Prebuild processing for globals.h and build.opt file', + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=textwrap.dedent(extra_txt)) + parser.add_argument('runtime_ide_path', help='Runtime IDE path, {runtime.ide.path}') + parser.add_argument('runtime_ide_version', type=int, help='Runtime IDE Version, {runtime.ide.version}') + parser.add_argument('build_path', help='Build path, {build.path}') + parser.add_argument('build_opt_fqfn', help="Build FQFN to build.opt") + parser.add_argument('source_globals_h_fqfn', help="Source FQFN Sketch.ino.globals.h") + parser.add_argument('commonhfile_fqfn', help="Core Source FQFN CommonHFile.h") + group = parser.add_mutually_exclusive_group(required=False) + group.add_argument('--cache_core', action='store_true', default=None, help='Assume a "compiler.cache_core" value of true') + group.add_argument('--no_cache_core', dest='cache_core', action='store_false', help='Assume a "compiler.cache_core" value of false') + group.add_argument('--preferences_file', help='Full path to preferences file') + group.add_argument('--preferences_sketch', nargs='?', action='store', const="preferences.txt", help='Sketch relative path to preferences file') + if "Linux" == platform.system(): + group.add_argument('--preferences_env', nargs='?', action='store', const="ARDUINO15_PREFERENCES_FILE", help='Use environment variable for path to preferences file') + return parser.parse_args() + # ref epilog, https://stackoverflow.com/a/50021771 + # ref nargs='*'', https://stackoverflow.com/a/4480202 + # ref no '--n' parameter, https://stackoverflow.com/a/21998252 def main(): global build_opt_signature global docs_url num_include_lines = 1 - if len(sys.argv) >= 6: - runtime_ide_path = os.path.normpath(sys.argv[1]) - build_path = os.path.normpath(sys.argv[2]) - build_opt_fqfn = os.path.normpath(sys.argv[3]) - source_globals_h_fqfn = os.path.normpath(sys.argv[4]) - commonhfile_fqfn = os.path.normpath(sys.argv[5]) + args = parse_args() + runtime_ide_path = os.path.normpath(args.runtime_ide_path) + build_path = os.path.normpath(args.build_path) + build_opt_fqfn = os.path.normpath(args.build_opt_fqfn) + source_globals_h_fqfn = os.path.normpath(args.source_globals_h_fqfn) + commonhfile_fqfn = os.path.normpath(args.commonhfile_fqfn) + if commonhfile_fqfn != None and len(commonhfile_fqfn): globals_name = os.path.basename(source_globals_h_fqfn) build_path_core, build_opt_name = os.path.split(build_opt_fqfn) globals_h_fqfn = os.path.join(build_path_core, globals_name) first_time = discover_1st_time_run(build_path) - use_aggressive_caching_workaround = check_preferences_txt(runtime_ide_path) + + use_aggressive_caching_workaround = determine_cache_state(args, runtime_ide_path, source_globals_h_fqfn) if first_time or \ not use_aggressive_caching_workaround or \ not os.path.exists(commonhfile_fqfn): enable_override(False, commonhfile_fqfn) + # A future timestamp on commonhfile_fqfn will cause everything to + # rebuild. This occurred during development and may happen after + # changing the system time. if time.time_ns() < os.stat(commonhfile_fqfn).st_mtime_ns: - print_err(f"Neutralize future timestamp on build file: {commonhfile_fqfn}") touch(commonhfile_fqfn) + print_err(f"Neutralized future timestamp on build file: {commonhfile_fqfn}") if not os.path.exists(build_path_core): os.makedirs(build_path_core) @@ -561,11 +653,11 @@ def main(): # touching commonhfile_fqfn in the source core tree will cause rebuild. # Looks like touching or writing unrelated files in the source core tree will cause rebuild. synchronous_touch(globals_h_fqfn, commonhfile_fqfn) - print_msg("Using 'aggressive caching' workaround.") + print_msg("Using 'aggressive caching' workaround, rebuild shared 'core.a' for current globals.") elif os.path.getsize(globals_h_fqfn): enable_override(True, commonhfile_fqfn) synchronous_touch(globals_h_fqfn, commonhfile_fqfn) - print_msg("Using 'aggressive caching' workaround.") + print_msg("Using 'aggressive caching' workaround, rebuild shared 'core.a' for current globals.") add_include_line(build_opt_fqfn, commonhfile_fqfn) add_include_line(build_opt_fqfn, globals_h_fqfn) @@ -586,8 +678,9 @@ def main(): print_msg(" Read more at " + docs_url) else: - print_err("Too few arguments. Add arguments:") - print_err(" Runtime IDE path, Build path, Build FQFN build.opt, Source FQFN Sketch.ino.globals.h, Core Source FQFN CommonHFile.h") + print_err(parser.parse_args('-h'.split())) + # print_err("Too few arguments. Add arguments:") + # print_err(" Runtime IDE path, Build path, Build FQFN build.opt, Source FQFN Sketch.ino.globals.h, Core Source FQFN CommonHFile.h") handle_error(1) handle_error(0) # commit print buffer From b75904ee36609c8b5653f6bd4a729a03bec71990 Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Thu, 24 Mar 2022 16:13:16 -0700 Subject: [PATCH 19/28] Fix script failure under windows Rely on argpaser for checking that all arguments are present. Removed redundant argument check in main(). Added '--debug' option and print_dbg method. Rethink failures on overrides. Remove well know path fallbacks, error exit when override file is missing. In well-known path search for preferences.txt, do not assume true. Make failure to find an error exit event. When Windows has two preferences.txt files and they have different values for caching and globals.h is used, error exit. It is not possible to know from the script which is being used. --- tools/mkbuildoptglobals.py | 352 ++++++++++++++++++++++++------------- 1 file changed, 232 insertions(+), 120 deletions(-) mode change 100755 => 100644 tools/mkbuildoptglobals.py diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py old mode 100755 new mode 100644 index 6e3895b88e..687088fbe4 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -78,8 +78,9 @@ globals.h.source.fqfn={build.source.path}/{build.project_name}.globals.h commonhfile.fqfn={build.core.path}/CommonHFile.h build.opt.fqfn={build.path}/core/build.opt +mkbuildoptglobals.extra_flags= -recipe.hooks.prebuild.2.pattern="{runtime.tools.python3.path}/python3" "{runtime.tools.mkbuildoptglobals}" "{runtime.ide.path}" "{build.path}" "{build.opt.fqfn}" "{globals.h.source.fqfn}" "{commonhfile.fqfn}" +recipe.hooks.prebuild.2.pattern="{runtime.tools.python3.path}/python3" "{runtime.tools.mkbuildoptglobals}" "{runtime.ide.path}" {runtime.ide.version} "{build.path}" "{build.opt.fqfn}" "{globals.h.source.fqfn}" "{commonhfile.fqfn}" {mkbuildoptglobals.extra_flags} compiler.cpreprocessor.flags=-D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -D_GNU_SOURCE -DESP8266 @{build.opt.path} "-I{compiler.sdk.path}/include" "-I{compiler.sdk.path}/{build.lwip_include}" "-I{compiler.libc.path}/include" "-I{build.path}/core" """ @@ -198,6 +199,7 @@ err_print_flag = False msg_print_buf = "" +debug_enabled = False # Issues trying to address through buffered printing # 1. Arduino IDE 2.0 RC5 does not show stderr text in color. Text printed does @@ -238,6 +240,13 @@ def print_err(*args, **kwargs): print_msg("***", *args, **kwargs) err_print_flag = True +def print_dbg(*args, **kwargs): + global debug_enabled + global err_print_flag + if debug_enabled: + print_msg("DEBUG:", *args, **kwargs) + err_print_flag = True + def handle_error(err_no): # on err_no 0, commit print buffer to stderr or stdout @@ -405,9 +414,19 @@ def discover_1st_time_run(build_path): def find_preferences_txt(runtime_ide_path): + """ + Check for perferences.txt in well-known locations. Most OSs have two + possibilities. When "portable" is present, it takes priority. Otherwise, the + remaining path wins. However, Windows has two. Depending on the install + source, the APP store or website download, both may appear and create an + ambiguous result. + + Return two item list - Two non "None" items indicate an ambiguous state. + + OS Path list for Arduino IDE 1.6.0 and newer + from: https://www.arduino.cc/en/hacking/preferences + """ platform_name = platform.system() - # OS Path list for Arduino IDE 1.6.0 and newer - # from: https://www.arduino.cc/en/hacking/preferences if "Linux" == platform_name: # Test for portable 1ST # /portable/preferences.txt (when used in portable mode) @@ -415,16 +434,16 @@ def find_preferences_txt(runtime_ide_path): fqfn = os.path.normpath(runtime_ide_path + "/portable/preferences.txt") # Linux - verified with Arduino IDE 1.8.19 if os.path.exists(fqfn): - return fqfn + return [fqfn, None] fqfn = os.path.expanduser("~/.arduino15/preferences.txt") # Linux - verified with Arduino IDE 1.8.18 and 2.0 RC5 64bit and AppImage if os.path.exists(fqfn): - return fqfn + return [fqfn, None] elif "Windows" == platform_name: fqfn = os.path.normpath(runtime_ide_path + "\portable\preferences.txt") # verified on Windows 10 with Arduino IDE 1.8.19 if os.path.exists(fqfn): - return fqfn + return [fqfn, None] # It is never simple. Arduino from the Windows APP store or the download # Windows 8 and up option will save "preferences.txt" in one location. # The downloaded Windows 7 (and up version) will put "preferences.txt" @@ -439,24 +458,27 @@ def find_preferences_txt(runtime_ide_path): print_err("Multiple 'preferences.txt' files found:") print_err(" " + fqfn) print_err(" " + fqfn2) - return fqfn + return [fqfn, None] else: - return fqfn + return [fqfn, fqfn2] elif os.path.exists(fqfn2): - return fqfn2 + return [fqfn2, None] elif "Darwin" == platform_name: # Portable is not compatable with Mac OS X # see https://docs.arduino.cc/software/ide-v1/tutorials/PortableIDE fqfn = os.path.expanduser("~/Library/Arduino15/preferences.txt") # Mac OS X - unverified if os.path.exists(fqfn): - return fqfn + return [fqfn, None] print_err("File preferences.txt not found on " + platform_name) - return "" + return [None, None] def get_preferences_txt(file_fqfn, key): + # Get Key Value, key is allowed to be missing. + # We assume file file_fqfn exists + basename = os.path.basename(file_fqfn) with open(file_fqfn) as file: for line in file: name, value = line.partition("=")[::2] @@ -464,24 +486,42 @@ def get_preferences_txt(file_fqfn, key): val = value.strip().lower() if val != 'true': val = False - print_msg(f" preferences.txt: {key}={val}") + print_msg(f" {basename}: {key}={val}") return val - print_err(" Key " + key + " not found in preferences.txt. Default to true.") + print_err(f" Key '{key}' not found in file {basename}. Default to true.") return True # If we don't find it just assume it is set True def check_preferences_txt(runtime_ide_path, preferences_file): + key = "compiler.cache_core" # return the state of "compiler.cache_core" found in preferences.txt - file_fqfn = preferences_file - if file_fqfn != None and os.path.exists(file_fqfn): - pass - else: + if preferences_file != None: + if os.path.exists(preferences_file): + print_msg(f"Using preferences from '{preferences_file}'") + return get_preferences_txt(preferences_file, key) + else: + print_err(f"Override preferences file '{preferences_file}' not found.") + + elif runtime_ide_path != None: + # For a particular install, search the expected locations for platform.txt + # This should never fail. file_fqfn = find_preferences_txt(runtime_ide_path) - if file_fqfn == "": - return True # cannot find file - assume enabled + if file_fqfn[0] != None: + print_msg(f"Using preferences from '{file_fqfn[0]}'") + val0 = get_preferences_txt(file_fqfn[0], key) + val1 = val0 + if file_fqfn[1] != None: + val1 = get_preferences_txt(file_fqfn[1], key) + if val0 == val1: # We can safely ignore that there were two preferences.txt files + return val0 + else: + print_err(f"Found too many preferences.txt files with different values for '{key}'") + raise UserWarning + else: + # Something is wrong with the installation or our understanding of the installation. + print_err("'preferences.txt' file missing from well known locations.") - print_msg("Using preferences from " + file_fqfn) - return get_preferences_txt(file_fqfn, "compiler.cache_core") + return None def touch(fname, times=None): @@ -497,29 +537,75 @@ def synchronous_touch(globals_h_fqfn, commonhfile_fqfn): with open(commonhfile_fqfn, 'a'): os.utime(commonhfile_fqfn, ns=(ts.st_atime_ns, ts.st_mtime_ns)) + def determine_cache_state(args, runtime_ide_path, source_globals_h_fqfn): - if args.runtime_ide_version < 10802: + global docs_url + print_dbg(f"runtime_ide_version: {args.runtime_ide_version}") + if args.runtime_ide_version < 10802: # CI also has version 10607 -- and args.runtime_ide_version != 10607: + # Aggresive core caching - not implemented before version 1.8.2 + # Note, Arduino IDE 2.0 rc5 has version 1.6.7 and has aggressive caching. + print_dbg(f"Old version ({args.runtime_ide_version}) of Arduino IDE no aggressive caching option") return False elif args.cache_core != None: print_msg(f"Preferences override, this prebuild script assumes the 'compiler.cache_core' parameter is set to {args.cache_core}") print_msg(f"To change, modify 'mkbuildoptglobals.extra_flags=(--cache_core | --no_cache_core)' in 'platform.local.txt'") return args.cache_core else: + ide_path = None preferences_fqfn = None - if args.preferences_file != None: - preferences_fqfn = args.preferences_file - elif args.preferences_sketch != None: - preferences_fqfn = os.path.normpath( - os.path.join( - os.path.dirname(source_globals_h_fqfn), - args.preferences_sketch)) - elif args.preferences_env != None: - preferences_fqfn = os.getenv(args.preferences_env) - return check_preferences_txt(runtime_ide_path, preferences_fqfn) + if args.preferences_sketch != None: + preferences_fqfn = os.path.join( + os.path.dirname(source_globals_h_fqfn), + os.path.normpath(args.preferences_sketch)) + else: + if args.preferences_file != None: + preferences_fqfn = args.preferences_file + elif args.preferences_env != None: + preferences_fqfn = args.preferences_env + else: + ide_path = runtime_ide_path + + if preferences_fqfn != None: + preferences_fqfn = os.path.normpath(preferences_fqfn) + root = False + if 'Windows' == platform.system(): + if preferences_fqfn[1:2] == ':\\': + root = True + else: + if preferences_fqfn[0] == '/': + root = True + if not root: + if preferences_fqfn[0] != '~': + preferences_fqfn = os.path.join("~", preferences_fqfn) + preferences_fqfn = os.path.expanduser(preferences_fqfn) + print_dbg(f"determine_cache_state: preferences_fqfn: {preferences_fqfn}") + + try: + caching_enabled = check_preferences_txt(ide_path, preferences_fqfn) + except UserWarning: + if os.path.exists(source_globals_h_fqfn): + caching_enabled = None + print_err(f" runtime_ide_version: {args.runtime_ide_version}") + print_err(f" This must be resolved to use '{globals_name}'") + print_err(f" Read more at {docs_url}") + else: + # We can quietly ignore the problem because we are not needed. + caching_enabled = True + + return caching_enabled """ -TODO sort out which of these are viable solutions +TODO + +aggressive caching workaround +========== ======= ========== +The question needs to be asked, is it a good idea? +With all this effort to aid in determining the cache state, it is rendered +usless when arduino command line switches are used that contradict our +settings. + +Sort out which of these are imperfect solutions should stay in Possible options for handling problems caused by: ./arduino --preferences-file other-preferences.txt @@ -548,6 +634,18 @@ def determine_cache_state(args, runtime_ide_path, source_globals_h_fqfn): """ +def check_env(env): + system = platform.system() + val = os.getenv(env) + if val == None: + if "Linux" == system or "Windows" == system: + raise argparse.ArgumentTypeError(f'Missing environment variable: {env}') + else: + # OS/Library limitation + raise argparse.ArgumentTypeError('Not supported') + return val + + def parse_args(): extra_txt = '''\ Use platform.local.txt 'mkbuildoptglobals.extra_flags=...' to supply override options: @@ -565,13 +663,15 @@ def parse_args(): parser.add_argument('build_opt_fqfn', help="Build FQFN to build.opt") parser.add_argument('source_globals_h_fqfn', help="Source FQFN Sketch.ino.globals.h") parser.add_argument('commonhfile_fqfn', help="Core Source FQFN CommonHFile.h") + parser.add_argument('--debug', action='store_true', required=False, default=False) group = parser.add_mutually_exclusive_group(required=False) group.add_argument('--cache_core', action='store_true', default=None, help='Assume a "compiler.cache_core" value of true') group.add_argument('--no_cache_core', dest='cache_core', action='store_false', help='Assume a "compiler.cache_core" value of false') group.add_argument('--preferences_file', help='Full path to preferences file') group.add_argument('--preferences_sketch', nargs='?', action='store', const="preferences.txt", help='Sketch relative path to preferences file') - if "Linux" == platform.system(): - group.add_argument('--preferences_env', nargs='?', action='store', const="ARDUINO15_PREFERENCES_FILE", help='Use environment variable for path to preferences file') + # Since the docs say most versions of Windows and Linux support the os.getenv method, suppress the help message. + group.add_argument('--preferences_env', nargs='?', action='store', type=check_env, const="ARDUINO15_PREFERENCES_FILE", help=argparse.SUPPRESS) + # ..., help='Use environment variable for path to preferences file') return parser.parse_args() # ref epilog, https://stackoverflow.com/a/50021771 # ref nargs='*'', https://stackoverflow.com/a/4480202 @@ -580,108 +680,120 @@ def parse_args(): def main(): global build_opt_signature global docs_url + global debug_enabled num_include_lines = 1 args = parse_args() + debug_enabled = args.debug runtime_ide_path = os.path.normpath(args.runtime_ide_path) build_path = os.path.normpath(args.build_path) build_opt_fqfn = os.path.normpath(args.build_opt_fqfn) source_globals_h_fqfn = os.path.normpath(args.source_globals_h_fqfn) commonhfile_fqfn = os.path.normpath(args.commonhfile_fqfn) - if commonhfile_fqfn != None and len(commonhfile_fqfn): - globals_name = os.path.basename(source_globals_h_fqfn) - build_path_core, build_opt_name = os.path.split(build_opt_fqfn) - globals_h_fqfn = os.path.join(build_path_core, globals_name) - - first_time = discover_1st_time_run(build_path) + globals_name = os.path.basename(source_globals_h_fqfn) + build_path_core, build_opt_name = os.path.split(build_opt_fqfn) + globals_h_fqfn = os.path.join(build_path_core, globals_name) - use_aggressive_caching_workaround = determine_cache_state(args, runtime_ide_path, source_globals_h_fqfn) + first_time = discover_1st_time_run(build_path) + if first_time: + print_dbg("First run since Arduino IDE started.") - if first_time or \ - not use_aggressive_caching_workaround or \ - not os.path.exists(commonhfile_fqfn): - enable_override(False, commonhfile_fqfn) - - # A future timestamp on commonhfile_fqfn will cause everything to - # rebuild. This occurred during development and may happen after - # changing the system time. - if time.time_ns() < os.stat(commonhfile_fqfn).st_mtime_ns: - touch(commonhfile_fqfn) - print_err(f"Neutralized future timestamp on build file: {commonhfile_fqfn}") - - if not os.path.exists(build_path_core): - os.makedirs(build_path_core) - print_msg("Clean build, created dir " + build_path_core) + use_aggressive_caching_workaround = determine_cache_state(args, runtime_ide_path, source_globals_h_fqfn) + if use_aggressive_caching_workaround == None: + # Specific rrror messages already buffered + handle_error(1) - if os.path.exists(source_globals_h_fqfn): - print_msg("Using global include from " + source_globals_h_fqfn) - - copy_create_build_file(source_globals_h_fqfn, globals_h_fqfn) - - # globals_h_fqfn timestamp was only updated if the source changed. This - # controls the rebuild on change. We can always extract a new build.opt - # w/o triggering a needless rebuild. - embedded_options = extract_create_build_opt_file(globals_h_fqfn, globals_name, build_opt_fqfn) - - if use_aggressive_caching_workaround: - # commonhfile_fqfn encodes the following information - # 1. When touched, it causes a rebuild of core.a - # 2. When file size is non-zero, it indicates we are using the - # aggressive cache workaround. The workaround is set to true - # (active) when we discover a non-zero length global .h file in - # any sketch. The aggressive workaround is cleared on the 1ST - # compile by the Arduino IDE after starting. - # 3. When the timestamp matches the build copy of globals.h - # (globals_h_fqfn), we know one two things: - # * The cached core.a matches up to the current build.opt and - # globals.h. The current sketch owns the cached copy of core.a. - # * globals.h has not changed, and no need to rebuild core.a - # 4. When core.a's timestamp does not match the build copy of - # the global .h file, we only know we need to rebuild core.a, and - # that is enough. - # - # When the sketch build has a "Sketch.ino.globals.h" file in the - # build tree that exactly matches the timestamp of "CommonHFile.h" - # in the platform source tree, it owns the core.a cache copy. If - # not, or "Sketch.ino.globals.h" has changed, rebuild core. - # A non-zero file size for commonhfile_fqfn, means we have seen a - # globals.h file before and workaround is active. - if os.path.getsize(commonhfile_fqfn): - if (os.path.getmtime(globals_h_fqfn) != os.path.getmtime(commonhfile_fqfn)): - # Need to rebuild core.a - # touching commonhfile_fqfn in the source core tree will cause rebuild. - # Looks like touching or writing unrelated files in the source core tree will cause rebuild. - synchronous_touch(globals_h_fqfn, commonhfile_fqfn) - print_msg("Using 'aggressive caching' workaround, rebuild shared 'core.a' for current globals.") - elif os.path.getsize(globals_h_fqfn): - enable_override(True, commonhfile_fqfn) + if first_time or \ + not use_aggressive_caching_workaround or \ + not os.path.exists(commonhfile_fqfn): + enable_override(False, commonhfile_fqfn) + + # A future timestamp on commonhfile_fqfn will cause everything to + # rebuild. This occurred during development and may happen after + # changing the system time. + if time.time_ns() < os.stat(commonhfile_fqfn).st_mtime_ns: + touch(commonhfile_fqfn) + print_err(f"Neutralized future timestamp on build file: {commonhfile_fqfn}") + + if not os.path.exists(build_path_core): + os.makedirs(build_path_core) + print_msg("Clean build, created dir " + build_path_core) + + if os.path.exists(source_globals_h_fqfn): + print_msg("Using global include from " + source_globals_h_fqfn) + + copy_create_build_file(source_globals_h_fqfn, globals_h_fqfn) + + # globals_h_fqfn timestamp was only updated if the source changed. This + # controls the rebuild on change. We can always extract a new build.opt + # w/o triggering a needless rebuild. + embedded_options = extract_create_build_opt_file(globals_h_fqfn, globals_name, build_opt_fqfn) + + if use_aggressive_caching_workaround: + # commonhfile_fqfn encodes the following information + # 1. When touched, it causes a rebuild of core.a + # 2. When file size is non-zero, it indicates we are using the + # aggressive cache workaround. The workaround is set to true + # (active) when we discover a non-zero length global .h file in + # any sketch. The aggressive workaround is cleared on the 1ST + # compile by the Arduino IDE after starting. + # 3. When the timestamp matches the build copy of globals.h + # (globals_h_fqfn), we know one two things: + # * The cached core.a matches up to the current build.opt and + # globals.h. The current sketch owns the cached copy of core.a. + # * globals.h has not changed, and no need to rebuild core.a + # 4. When core.a's timestamp does not match the build copy of + # the global .h file, we only know we need to rebuild core.a, and + # that is enough. + # + # When the sketch build has a "Sketch.ino.globals.h" file in the + # build tree that exactly matches the timestamp of "CommonHFile.h" + # in the platform source tree, it owns the core.a cache copy. If + # not, or "Sketch.ino.globals.h" has changed, rebuild core. + # A non-zero file size for commonhfile_fqfn, means we have seen a + # globals.h file before and workaround is active. + if debug_enabled: + ts = os.stat(globals_h_fqfn) + print_dbg(f"globals_h_fqfn ns_stamp = {ts.st_mtime_ns}") + print_dbg(f"getmtime(globals_h_fqfn) {os.path.getmtime(globals_h_fqfn)}") + ts = os.stat(commonhfile_fqfn) + print_dbg(f"commonhfile_fqfn ns_stamp = {ts.st_mtime_ns}") + print_dbg(f"getmtime(commonhfile_fqfn) {os.path.getmtime(commonhfile_fqfn)}") + + if os.path.getsize(commonhfile_fqfn): + if (os.path.getmtime(globals_h_fqfn) != os.path.getmtime(commonhfile_fqfn)): + # Need to rebuild core.a + # touching commonhfile_fqfn in the source core tree will cause rebuild. + # Looks like touching or writing unrelated files in the source core tree will cause rebuild. synchronous_touch(globals_h_fqfn, commonhfile_fqfn) print_msg("Using 'aggressive caching' workaround, rebuild shared 'core.a' for current globals.") - - add_include_line(build_opt_fqfn, commonhfile_fqfn) - add_include_line(build_opt_fqfn, globals_h_fqfn) - - # Provide context help for build option support. - source_build_opt_h_fqfn = os.path.join(os.path.dirname(source_globals_h_fqfn), "build_opt.h") - if os.path.exists(source_build_opt_h_fqfn) and not embedded_options: - print_err("Build options file '" + source_build_opt_h_fqfn + "' not supported.") - print_err(" Add build option content to '" + source_globals_h_fqfn + "'.") - print_err(" Embedd compiler command-line options in a block comment starting with '" + build_opt_signature + "'.") - print_err(" Read more at " + docs_url) - elif os.path.exists(source_globals_h_fqfn): - if not embedded_options: - print_msg("Tip: Embedd compiler command-line options in a block comment starting with '" + build_opt_signature + "'.") - print_msg(" Read more at " + docs_url) + else: + print_dbg(f"Using old cached 'core.a'") + elif os.path.getsize(globals_h_fqfn): + enable_override(True, commonhfile_fqfn) + synchronous_touch(globals_h_fqfn, commonhfile_fqfn) + print_msg("Using 'aggressive caching' workaround, rebuild shared 'core.a' for current globals.") else: - print_msg("Note: optional global include file '" + source_globals_h_fqfn + "' does not exist.") + print_dbg(f"Workaround not active/needed") + + add_include_line(build_opt_fqfn, commonhfile_fqfn) + add_include_line(build_opt_fqfn, globals_h_fqfn) + + # Provide context help for build option support. + source_build_opt_h_fqfn = os.path.join(os.path.dirname(source_globals_h_fqfn), "build_opt.h") + if os.path.exists(source_build_opt_h_fqfn) and not embedded_options: + print_err("Build options file '" + source_build_opt_h_fqfn + "' not supported.") + print_err(" Add build option content to '" + source_globals_h_fqfn + "'.") + print_err(" Embedd compiler command-line options in a block comment starting with '" + build_opt_signature + "'.") + print_err(" Read more at " + docs_url) + elif os.path.exists(source_globals_h_fqfn): + if not embedded_options: + print_msg("Tip: Embedd compiler command-line options in a block comment starting with '" + build_opt_signature + "'.") print_msg(" Read more at " + docs_url) - else: - print_err(parser.parse_args('-h'.split())) - # print_err("Too few arguments. Add arguments:") - # print_err(" Runtime IDE path, Build path, Build FQFN build.opt, Source FQFN Sketch.ino.globals.h, Core Source FQFN CommonHFile.h") - handle_error(1) + print_msg("Note: optional global include file '" + source_globals_h_fqfn + "' does not exist.") + print_msg(" Read more at " + docs_url) handle_error(0) # commit print buffer From 75d241e2cd0b93786d1dcf65580dd4d3058ff6bd Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Tue, 29 Mar 2022 15:19:04 -0700 Subject: [PATCH 20/28] Use quotes on build.opt Update comment Include the @ within the expantion string use quotes around file name. Update doc example to remind and use quotes. --- doc/faq/a06-global-build-options.rst | 8 +++++--- platform.txt | 3 ++- tools/mkbuildoptglobals.py | 7 ++++++- 3 files changed, 13 insertions(+), 5 deletions(-) diff --git a/doc/faq/a06-global-build-options.rst b/doc/faq/a06-global-build-options.rst index 0e65e526f7..e2996114a4 100644 --- a/doc/faq/a06-global-build-options.rst +++ b/doc/faq/a06-global-build-options.rst @@ -197,11 +197,13 @@ Examples of hints: :: - mkbuildoptglobals.extra_flags=--preferences_sketch # assume file preferences.txt in the sketch folder - mkbuildoptglobals.extra_flags=--preferences_sketch pref.txt # is relative to the sketch folder + mkbuildoptglobals.extra_flags=--preferences_sketch # assume file preferences.txt in the sketch folder + mkbuildoptglobals.extra_flags=--preferences_sketch "pref.txt" # is relative to the sketch folder mkbuildoptglobals.extra_flags=--no_cache_core mkbuildoptglobals.extra_flags=--cache_core - mkbuildoptglobals.extra_flags=--preferences_file other-preferences.txt # relative to IDE or full path + mkbuildoptglobals.extra_flags=--preferences_file "other-preferences.txt" # relative to IDE or full path + +If required, remember to quote file or file paths. **Multiple versions of the Arduino IDE running** diff --git a/platform.txt b/platform.txt index 5295aa62f3..3211795a3d 100644 --- a/platform.txt +++ b/platform.txt @@ -63,13 +63,14 @@ build.spiffs_blocksize= globals.h.source.fqfn={build.source.path}/{build.project_name}.globals.h commonhfile.fqfn={build.core.path}/CommonHFile.h build.opt.fqfn={build.path}/core/build.opt +build.opt.flags="@{build.opt.fqfn}" mkbuildoptglobals.extra_flags= compiler.path={runtime.tools.xtensa-lx106-elf-gcc.path}/bin/ compiler.sdk.path={runtime.platform.path}/tools/sdk compiler.libc.path={runtime.platform.path}/tools/sdk/libc/xtensa-lx106-elf -compiler.cpreprocessor.flags=-D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -D_GNU_SOURCE -DESP8266 @{build.opt.fqfn} "-I{compiler.sdk.path}/include" "-I{compiler.sdk.path}/{build.lwip_include}" "-I{compiler.libc.path}/include" "-I{build.path}/core" +compiler.cpreprocessor.flags=-D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__ -D_GNU_SOURCE -DESP8266 {build.opt.flags} "-I{compiler.sdk.path}/include" "-I{compiler.sdk.path}/{build.lwip_include}" "-I{compiler.libc.path}/include" "-I{build.path}/core" # support precompiled libraries in IDE v1.8.6+ compiler.libraries.ldflags= diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index 687088fbe4..8967f97176 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -615,7 +615,7 @@ def determine_cache_state(args, runtime_ide_path, source_globals_h_fqfn): --no_cache_core --preferences_file (relative to IDE or full path) --preferences_sketch (default looks for preferences.txt or specify path relative to sketch folder) ---preferences_env, only works on Linux +--preferences_env, python docs say "Availability: most flavors of Unix, Windows." export ARDUINO15_PREFERENCES_FILE=$(realpath other-name-than-default-preferences.txt ) ./arduino --preferences-file other-name-than-default-preferences.txt @@ -636,6 +636,11 @@ def determine_cache_state(args, runtime_ide_path, source_globals_h_fqfn): def check_env(env): system = platform.system() + # From the docs: + # Availability: most flavors of Unix, Windows. + # “Availability: Unix” are supported on macOS + # Because of the soft commitment, I used "help=argparse.SUPPRESS" to keep + # the claim out of the help. The unavailable case is untested. val = os.getenv(env) if val == None: if "Linux" == system or "Windows" == system: From b6cc66dfd142b38ddd6d5aa4bb48c10791a08eee Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Wed, 30 Mar 2022 11:49:23 -0700 Subject: [PATCH 21/28] Update CI for build option and global support Added "mkbuildoptglobals.extra_flags=--cache_core" to platform.loca.txt Update "-ide-version=10802" this version number indicates aggressive caching support Added example to test global .h support --- .../GlobalBuildOptions/GlobalBuildOptions.ino | 31 +++++++++++++++ .../GlobalBuildOptions.ino.globals.h | 39 +++++++++++++++++++ tests/common.sh | 2 +- tools/build.py | 2 +- 4 files changed, 72 insertions(+), 2 deletions(-) create mode 100644 libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino create mode 100644 libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino.globals.h diff --git a/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino b/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino new file mode 100644 index 0000000000..2b47dac412 --- /dev/null +++ b/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino @@ -0,0 +1,31 @@ +/* + * Showcase the use of embedded build options and global defines through a specially named .h file. + * Sketch file name followed by ".globals.h", "GlobalBuildOptions.ino.globals.h" + * + * Example from https://arduino-esp8266.readthedocs.io/en/latest/faq/a06-global-build-options.html + * + * Note, we do not "#include" the special file "GlobalBuildOptions.ino.globals.h". + * The prebuild script will make it available to all modules. + * + * To track the new sketch name when saving this sketch to a new location and + * name, remember to update the global .h file name. + */ + +#include // has prototype for umm_free_heap_size_min() + +void setup() { + Serial.begin(115200); + delay(200); + +#ifdef MYTITLE1 + Serial.printf("\r\n" MYTITLE1 MYTITLE2 "\r\n"); +#else + Serial.println("ERROR: MYTITLE1 not present"); +#endif + +#ifdef UMM_STATS_FULL + Serial.printf("Heap Low Watermark %u\r\n", umm_free_heap_size_min()); +#endif +} + +void loop() {} diff --git a/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino.globals.h b/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino.globals.h new file mode 100644 index 0000000000..34e4f24664 --- /dev/null +++ b/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino.globals.h @@ -0,0 +1,39 @@ +/*@create-file:build.opt@ + // An embedded build.opt file using a "C" block comment. The starting signature + // must be on a line by itself. The closing block comment pattern should be on a + // line by itself. Each line within the block comment will be space trimmed and + // written to build.opt, skipping blank lines and lines starting with '//', '*' + // or '#'. + -DMYTITLE1="\"Running on \"" + * this line is ignored + *@create-file:build.opt@ + # this line is ignored + -O3 +// -fanalyzer + -DUMM_STATS_FULL=1 +*/ + +#ifndef GLOBALBUILDOPTIONS_INO_GLOBALS_H +#define GLOBALBUILDOPTIONS_INO_GLOBALS_H + +#if !defined(__ASSEMBLER__) +// Defines kept away from assembler modules +// i.e. Defines for .cpp, .ino, .c ... modules +#endif + +#if defined(__cplusplus) +// Defines kept private to .cpp and .ino modules +//#pragma message("__cplusplus has been seen") +#define MYTITLE2 "Empty" +#endif + +#if !defined(__cplusplus) && !defined(__ASSEMBLER__) +// Defines kept private to .c modules +#define MYTITLE2 "~Full" +#endif + +#if defined(__ASSEMBLER__) +// Defines kept private to assembler modules +#endif + +#endif diff --git a/tests/common.sh b/tests/common.sh index b6ce2fe61e..30e476a8ad 100755 --- a/tests/common.sh +++ b/tests/common.sh @@ -197,6 +197,7 @@ function install_ide() # Set custom warnings for all builds (i.e. could add -Wextra at some point) echo "compiler.c.extra_flags=-Wall -Wextra -Werror $debug_flags" > esp8266/platform.local.txt echo "compiler.cpp.extra_flags=-Wall -Wextra -Werror $debug_flags" >> esp8266/platform.local.txt + echo "mkbuildoptglobals.extra_flags=--cache_core" >> esp8266/platform.local.txt echo -e "\n----platform.local.txt----" cat esp8266/platform.local.txt echo -e "\n----\n" @@ -250,4 +251,3 @@ if [ -z "$TRAVIS_BUILD_DIR" ]; then popd > /dev/null echo "TRAVIS_BUILD_DIR=$TRAVIS_BUILD_DIR" fi - diff --git a/tools/build.py b/tools/build.py index 791ac30e2a..378c4a9f9d 100755 --- a/tools/build.py +++ b/tools/build.py @@ -73,7 +73,7 @@ def compile(tmp_dir, sketch, cache, tools_dir, hardware_dir, ide_path, f, args): fqbn += ',waveform=phase' cmd += [fqbn] cmd += ['-built-in-libraries', ide_path + '/libraries'] - cmd += ['-ide-version=10607'] + cmd += ['-ide-version=10802'] cmd += ['-warnings={warnings}'.format(**vars(args))] if args.verbose: cmd += ['-verbose'] From c4a6724f5eb499cc93dd8fb9d4e0ea866876ecb1 Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Wed, 30 Mar 2022 16:25:02 -0700 Subject: [PATCH 22/28] Add debug prints Added --debug to CI - this needs to be removed later Tweaks to touch... --- tests/common.sh | 2 +- tools/mkbuildoptglobals.py | 46 ++++++++++++++++++++++++++++---------- 2 files changed, 35 insertions(+), 13 deletions(-) diff --git a/tests/common.sh b/tests/common.sh index 30e476a8ad..5deb98fc39 100755 --- a/tests/common.sh +++ b/tests/common.sh @@ -197,7 +197,7 @@ function install_ide() # Set custom warnings for all builds (i.e. could add -Wextra at some point) echo "compiler.c.extra_flags=-Wall -Wextra -Werror $debug_flags" > esp8266/platform.local.txt echo "compiler.cpp.extra_flags=-Wall -Wextra -Werror $debug_flags" >> esp8266/platform.local.txt - echo "mkbuildoptglobals.extra_flags=--cache_core" >> esp8266/platform.local.txt + echo "mkbuildoptglobals.extra_flags=--debug --cache_core" >> esp8266/platform.local.txt echo -e "\n----platform.local.txt----" cat esp8266/platform.local.txt echo -e "\n----\n" diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index 8967f97176..66d470af66 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -525,18 +525,26 @@ def check_preferences_txt(runtime_ide_path, preferences_file): def touch(fname, times=None): - with open(fname, 'a'): - os.utime(fname, times) - + with open(fname, "a") as file: + os.utime(file.fileno(), times) def synchronous_touch(globals_h_fqfn, commonhfile_fqfn): + global debug_enabled # touch both files with the same timestamp - with open(globals_h_fqfn, 'a'): - os.utime(globals_h_fqfn) - ts = os.stat(globals_h_fqfn) - with open(commonhfile_fqfn, 'a'): - os.utime(commonhfile_fqfn, ns=(ts.st_atime_ns, ts.st_mtime_ns)) + touch(globals_h_fqfn) + with open(globals_h_fqfn, 'r') as file: + ts = os.stat(file.fileno()) + with open(commonhfile_fqfn, 'a') as file2: + os.utime(file2.fileno(), ns=(ts.st_atime_ns, ts.st_mtime_ns)) + if debug_enabled: + print_dbg("After synchronous_touch") + ts = os.stat(globals_h_fqfn) + print_dbg(f" globals_h_fqfn ns_stamp = {ts.st_mtime_ns}") + print_dbg(f" getmtime(globals_h_fqfn) {os.path.getmtime(globals_h_fqfn)}") + ts = os.stat(commonhfile_fqfn) + print_dbg(f" commonhfile_fqfn ns_stamp = {ts.st_mtime_ns}") + print_dbg(f" getmtime(commonhfile_fqfn) {os.path.getmtime(commonhfile_fqfn)}") def determine_cache_state(args, runtime_ide_path, source_globals_h_fqfn): global docs_url @@ -700,6 +708,16 @@ def main(): build_path_core, build_opt_name = os.path.split(build_opt_fqfn) globals_h_fqfn = os.path.join(build_path_core, globals_name) + print_dbg(f"runtime_ide_path: {runtime_ide_path}") + print_dbg(f"runtime_ide_version: {args.runtime_ide_version}") + print_dbg(f"build_path: {build_path}") + print_dbg(f"build_opt_fqfn: {build_opt_fqfn}") + print_dbg(f"source_globals_h_fqfn: {source_globals_h_fqfn}") + print_dbg(f"commonhfile_fqfn: {commonhfile_fqfn}") + print_dbg(f"globals_name: {globals_name}") + print_dbg(f"build_path_core: {build_path_core}") + print_dbg(f"globals_h_fqfn: {globals_h_fqfn}") + first_time = discover_1st_time_run(build_path) if first_time: print_dbg("First run since Arduino IDE started.") @@ -709,6 +727,9 @@ def main(): # Specific rrror messages already buffered handle_error(1) + print_dbg(f"first_time: {first_time}") + print_dbg(f"use_aggressive_caching_workaround: {use_aggressive_caching_workaround}") + if first_time or \ not use_aggressive_caching_workaround or \ not os.path.exists(commonhfile_fqfn): @@ -759,12 +780,13 @@ def main(): # A non-zero file size for commonhfile_fqfn, means we have seen a # globals.h file before and workaround is active. if debug_enabled: + print_dbg("Timestamps at start of check aggressive caching workaround") ts = os.stat(globals_h_fqfn) - print_dbg(f"globals_h_fqfn ns_stamp = {ts.st_mtime_ns}") - print_dbg(f"getmtime(globals_h_fqfn) {os.path.getmtime(globals_h_fqfn)}") + print_dbg(f" globals_h_fqfn ns_stamp = {ts.st_mtime_ns}") + print_dbg(f" getmtime(globals_h_fqfn) {os.path.getmtime(globals_h_fqfn)}") ts = os.stat(commonhfile_fqfn) - print_dbg(f"commonhfile_fqfn ns_stamp = {ts.st_mtime_ns}") - print_dbg(f"getmtime(commonhfile_fqfn) {os.path.getmtime(commonhfile_fqfn)}") + print_dbg(f" commonhfile_fqfn ns_stamp = {ts.st_mtime_ns}") + print_dbg(f" getmtime(commonhfile_fqfn) {os.path.getmtime(commonhfile_fqfn)}") if os.path.getsize(commonhfile_fqfn): if (os.path.getmtime(globals_h_fqfn) != os.path.getmtime(commonhfile_fqfn)): From b0eb49734d7179d4ff341ef50ccb83d9b0df2a06 Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Wed, 30 Mar 2022 19:05:34 -0700 Subject: [PATCH 23/28] Give each build VM a unique build.tmp space --- tests/common.sh | 2 +- tools/mkbuildoptglobals.py | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/common.sh b/tests/common.sh index 5deb98fc39..a0737d1540 100755 --- a/tests/common.sh +++ b/tests/common.sh @@ -54,9 +54,9 @@ function build_sketches() local arduino=$1 local srcpath=$2 local build_arg=$3 - local build_dir=build.tmp local build_mod=$4 local build_rem=$5 + local build_dir=build.tmp.$build_rem.$build_mod local lwip=$6 mkdir -p $build_dir local build_cmd="python3 tools/build.py -b generic -v -w all -s 4M1M -v -k --build_cache $cache_dir -p ./$build_dir -n $lwip $build_arg " diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index 66d470af66..7fc5936efe 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -276,11 +276,13 @@ def copy_create_build_file(source_fqfn, build_target_fqfn): if os.path.exists(build_target_fqfn) and \ os.path.getmtime(build_target_fqfn) >= os.path.getmtime(source_fqfn): # only copy newer files - do nothing, all is good + print_dbg(f"up to date os.path.exists({source_fqfn}) ") return False else: # The new copy gets stamped with the current time, just as other # files copied by `arduino-builder`. copyfile(source_fqfn, build_target_fqfn) + print_dbg(f"copyfile({source_fqfn}, {build_target_fqfn})") else: if os.path.exists(build_target_fqfn) and \ os.path.getsize(build_target_fqfn) == 0: From e76a1909a2695c021797ce3909e0d23ec9bdccca Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Thu, 31 Mar 2022 09:54:45 -0700 Subject: [PATCH 24/28] Corrected style on example temp CI changes debug crud Added --ci switch --- .../GlobalBuildOptions/GlobalBuildOptions.ino | 10 +++++----- tests/build.sh | 1 - tests/common.sh | 10 ++++++---- tools/mkbuildoptglobals.py | 14 +++++++++++--- 4 files changed, 22 insertions(+), 13 deletions(-) diff --git a/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino b/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino index 2b47dac412..7a37b8b67c 100644 --- a/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino +++ b/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino @@ -14,17 +14,17 @@ #include // has prototype for umm_free_heap_size_min() void setup() { - Serial.begin(115200); - delay(200); + Serial.begin(115200); + delay(200); #ifdef MYTITLE1 - Serial.printf("\r\n" MYTITLE1 MYTITLE2 "\r\n"); + Serial.printf("\r\n" MYTITLE1 MYTITLE2 "\r\n"); #else - Serial.println("ERROR: MYTITLE1 not present"); + Serial.println("ERROR: MYTITLE1 not present"); #endif #ifdef UMM_STATS_FULL - Serial.printf("Heap Low Watermark %u\r\n", umm_free_heap_size_min()); + Serial.printf("Heap Low Watermark %u\r\n", umm_free_heap_size_min()); #endif } diff --git a/tests/build.sh b/tests/build.sh index 45f88e7ef2..5f9e1be4cb 100755 --- a/tests/build.sh +++ b/tests/build.sh @@ -19,4 +19,3 @@ install_arduino nodebug build_sketches_with_arduino "$mod" "$rem" lm2f rm -rf "$cache_dir" - diff --git a/tests/common.sh b/tests/common.sh index a0737d1540..12fcbd73cc 100755 --- a/tests/common.sh +++ b/tests/common.sh @@ -57,9 +57,11 @@ function build_sketches() local build_mod=$4 local build_rem=$5 local build_dir=build.tmp.$build_rem.$build_mod + local lcache_dir=$cache_dir/vm$build_rem.$build_mod local lwip=$6 mkdir -p $build_dir - local build_cmd="python3 tools/build.py -b generic -v -w all -s 4M1M -v -k --build_cache $cache_dir -p ./$build_dir -n $lwip $build_arg " + mkdir -p $lcache_dir + local build_cmd="python3 tools/build.py -b generic -v -w all -s 4M1M -v -k --build_cache $lcache_dir -p ./$build_dir -n $lwip $build_arg " if [ "$WINDOWS" = "1" ]; then # Paths to the arduino builder need to be / referenced, not our native ones build_cmd=$(echo $build_cmd --ide_path $arduino | sed 's/ \/c\// \//g' ) # replace '/c/' with '/' @@ -74,14 +76,14 @@ function build_sketches() continue # Not ours to do fi - if [ -e $cache_dir/core/*.a ]; then + if [ -e $lcache_dir/core/*.a ]; then # We need to preserve the build.options.json file and replace the last .ino # with this sketch's ino file, or builder will throw everything away. jq '."sketchLocation" = "'$sketch'"' $build_dir/build.options.json > $build_dir/build.options.json.tmp mv $build_dir/build.options.json.tmp $build_dir/build.options.json # Set the time of the cached core.a file to the future so the GIT header # we regen won't cause the builder to throw it out and rebuild from scratch. - touch -d 'now + 1 day' $cache_dir/core/*.a + touch -d 'now + 1 day' $lcache_dir/core/*.a fi # Clear out the last built sketch, map, elf, bin files, but leave the compiled @@ -197,7 +199,7 @@ function install_ide() # Set custom warnings for all builds (i.e. could add -Wextra at some point) echo "compiler.c.extra_flags=-Wall -Wextra -Werror $debug_flags" > esp8266/platform.local.txt echo "compiler.cpp.extra_flags=-Wall -Wextra -Werror $debug_flags" >> esp8266/platform.local.txt - echo "mkbuildoptglobals.extra_flags=--debug --cache_core" >> esp8266/platform.local.txt + echo "mkbuildoptglobals.extra_flags=--debug --ci --cache_core" >> esp8266/platform.local.txt echo -e "\n----platform.local.txt----" cat esp8266/platform.local.txt echo -e "\n----\n" diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index 7fc5936efe..b39b3dc857 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -679,6 +679,7 @@ def parse_args(): parser.add_argument('source_globals_h_fqfn', help="Source FQFN Sketch.ino.globals.h") parser.add_argument('commonhfile_fqfn', help="Core Source FQFN CommonHFile.h") parser.add_argument('--debug', action='store_true', required=False, default=False) + parser.add_argument('--ci', action='store_true', required=False, default=False) group = parser.add_mutually_exclusive_group(required=False) group.add_argument('--cache_core', action='store_true', default=None, help='Assume a "compiler.cache_core" value of true') group.add_argument('--no_cache_core', dest='cache_core', action='store_false', help='Assume a "compiler.cache_core" value of false') @@ -720,9 +721,16 @@ def main(): print_dbg(f"build_path_core: {build_path_core}") print_dbg(f"globals_h_fqfn: {globals_h_fqfn}") - first_time = discover_1st_time_run(build_path) - if first_time: - print_dbg("First run since Arduino IDE started.") + if args.ci: + # Requires CommonHFile to never be checked in. + if os.path.exists(commonhfile_fqfn): + first_time = False + else: + first_time = True + else: + first_time = discover_1st_time_run(build_path) + if first_time: + print_dbg("First run since Arduino IDE started.") use_aggressive_caching_workaround = determine_cache_state(args, runtime_ide_path, source_globals_h_fqfn) if use_aggressive_caching_workaround == None: From f86c640bd20ac325c519b181f45c445470fc270e Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Thu, 31 Mar 2022 12:32:11 -0700 Subject: [PATCH 25/28] Removed CI debug crud run_CI_locall.sh works fine locally. Hosted Multi-VM CI fails to work with 'aggressive caching' workaround method. Add #if defined(CORE_MOCK) to failing example. --- .../GlobalBuildOptions/GlobalBuildOptions.ino | 2 +- tests/common.sh | 12 +++++------- tools/mkbuildoptglobals.py | 2 +- 3 files changed, 7 insertions(+), 9 deletions(-) diff --git a/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino b/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino index 7a37b8b67c..9aa13ce3b4 100644 --- a/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino +++ b/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino @@ -23,7 +23,7 @@ void setup() { Serial.println("ERROR: MYTITLE1 not present"); #endif -#ifdef UMM_STATS_FULL +#if defined(UMM_STATS_FULL) && !defined(CORE_MOCK) Serial.printf("Heap Low Watermark %u\r\n", umm_free_heap_size_min()); #endif } diff --git a/tests/common.sh b/tests/common.sh index 12fcbd73cc..5b273e2d14 100755 --- a/tests/common.sh +++ b/tests/common.sh @@ -54,14 +54,12 @@ function build_sketches() local arduino=$1 local srcpath=$2 local build_arg=$3 + local build_dir=build.tmp local build_mod=$4 local build_rem=$5 - local build_dir=build.tmp.$build_rem.$build_mod - local lcache_dir=$cache_dir/vm$build_rem.$build_mod local lwip=$6 mkdir -p $build_dir - mkdir -p $lcache_dir - local build_cmd="python3 tools/build.py -b generic -v -w all -s 4M1M -v -k --build_cache $lcache_dir -p ./$build_dir -n $lwip $build_arg " + local build_cmd="python3 tools/build.py -b generic -v -w all -s 4M1M -v -k --build_cache $cache_dir -p ./$build_dir -n $lwip $build_arg " if [ "$WINDOWS" = "1" ]; then # Paths to the arduino builder need to be / referenced, not our native ones build_cmd=$(echo $build_cmd --ide_path $arduino | sed 's/ \/c\// \//g' ) # replace '/c/' with '/' @@ -76,14 +74,14 @@ function build_sketches() continue # Not ours to do fi - if [ -e $lcache_dir/core/*.a ]; then + if [ -e $cache_dir/core/*.a ]; then # We need to preserve the build.options.json file and replace the last .ino # with this sketch's ino file, or builder will throw everything away. jq '."sketchLocation" = "'$sketch'"' $build_dir/build.options.json > $build_dir/build.options.json.tmp mv $build_dir/build.options.json.tmp $build_dir/build.options.json # Set the time of the cached core.a file to the future so the GIT header # we regen won't cause the builder to throw it out and rebuild from scratch. - touch -d 'now + 1 day' $lcache_dir/core/*.a + touch -d 'now + 1 day' $cache_dir/core/*.a fi # Clear out the last built sketch, map, elf, bin files, but leave the compiled @@ -199,7 +197,7 @@ function install_ide() # Set custom warnings for all builds (i.e. could add -Wextra at some point) echo "compiler.c.extra_flags=-Wall -Wextra -Werror $debug_flags" > esp8266/platform.local.txt echo "compiler.cpp.extra_flags=-Wall -Wextra -Werror $debug_flags" >> esp8266/platform.local.txt - echo "mkbuildoptglobals.extra_flags=--debug --ci --cache_core" >> esp8266/platform.local.txt + echo "mkbuildoptglobals.extra_flags=--ci --cache_core" >> esp8266/platform.local.txt echo -e "\n----platform.local.txt----" cat esp8266/platform.local.txt echo -e "\n----\n" diff --git a/tools/mkbuildoptglobals.py b/tools/mkbuildoptglobals.py index b39b3dc857..188b07b42a 100644 --- a/tools/mkbuildoptglobals.py +++ b/tools/mkbuildoptglobals.py @@ -722,7 +722,7 @@ def main(): print_dbg(f"globals_h_fqfn: {globals_h_fqfn}") if args.ci: - # Requires CommonHFile to never be checked in. + # Requires CommonHFile.h to never be checked in. if os.path.exists(commonhfile_fqfn): first_time = False else: From fad05ec0e43f7129f034b83b532ae780f51dfcb6 Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Thu, 31 Mar 2022 14:36:08 -0700 Subject: [PATCH 26/28] Try HOST_MOCK --- .../esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino b/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino index 9aa13ce3b4..e785ff2934 100644 --- a/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino +++ b/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino @@ -23,7 +23,7 @@ void setup() { Serial.println("ERROR: MYTITLE1 not present"); #endif -#if defined(UMM_STATS_FULL) && !defined(CORE_MOCK) +#if defined(UMM_STATS_FULL) && !defined(HOST_MOCK) Serial.printf("Heap Low Watermark %u\r\n", umm_free_heap_size_min()); #endif } From 29852c1f591be51ec9591fc86de7bdfef39391e4 Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Thu, 31 Mar 2022 20:27:35 -0700 Subject: [PATCH 27/28] CI adjustments mkbuildoptglobals.py is optimized around the Arduino IDE 1.x behaviour. One way the CI differs from the Arduino IDE is in the handling of core and caching core. With the Arduino IDE, each sketch has a private copy of core and contributes to a core cache. With the CI, there is one shared copy of core for all sketches. When global options are used, the shared copy of core and cache are removed before and after the build. --- .../GlobalBuildOptions/GlobalBuildOptions.ino | 2 +- .../GlobalBuildOptions.ino.globals.h | 4 +-- tests/common.sh | 33 +++++++++++++++++++ 3 files changed, 36 insertions(+), 3 deletions(-) diff --git a/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino b/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino index e785ff2934..2bc18751e1 100644 --- a/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino +++ b/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino @@ -23,7 +23,7 @@ void setup() { Serial.println("ERROR: MYTITLE1 not present"); #endif -#if defined(UMM_STATS_FULL) && !defined(HOST_MOCK) +#if defined(UMM_STATS_FULL) Serial.printf("Heap Low Watermark %u\r\n", umm_free_heap_size_min()); #endif } diff --git a/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino.globals.h b/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino.globals.h index 34e4f24664..12a4882ff8 100644 --- a/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino.globals.h +++ b/libraries/esp8266/examples/GlobalBuildOptions/GlobalBuildOptions.ino.globals.h @@ -8,8 +8,8 @@ * this line is ignored *@create-file:build.opt@ # this line is ignored - -O3 -// -fanalyzer + -O3 + // -fanalyzer -DUMM_STATS_FULL=1 */ diff --git a/tests/common.sh b/tests/common.sh index 5b273e2d14..257119ebb6 100755 --- a/tests/common.sh +++ b/tests/common.sh @@ -67,6 +67,8 @@ function build_sketches() local sketches=$(find $srcpath -name *.ino | sort) print_size_info >size.log export ARDUINO_IDE_PATH=$arduino + local k_partial_core_cleanup=("build.opt" "*.ino.globals.h") + local mk_clean_core=1 local testcnt=0 for sketch in $sketches; do testcnt=$(( ($testcnt + 1) % $build_mod )) @@ -74,6 +76,26 @@ function build_sketches() continue # Not ours to do fi + # mkbuildoptglobals.py is optimized around the Arduino IDE 1.x + # behaviour. One way the CI differs from the Arduino IDE is in the + # handling of core and caching core. With the Arduino IDE, each sketch + # has a private copy of core and contributes to a core cache. With the + # CI, there is one shared copy of core for all sketches. When global + # options are used, the shared copy of core and cache are removed before + # and after the build. + # + # Do we need a clean core build? $build_dir/core/* cannot be shared + # between sketches when global options are present. + if [ -s ${sketch}.globals.h ]; then + mk_clean_core=1 + fi + if [ $mk_clean_core -ne 0 ]; then + rm -rf rm $build_dir/core/* + else + # Remove sketch specific files from ./core/ between builds. + rm -rf $build_dir/core/build.opt $build_dir/core/*.ino.globals.h + fi + if [ -e $cache_dir/core/*.a ]; then # We need to preserve the build.options.json file and replace the last .ino # with this sketch's ino file, or builder will throw everything away. @@ -82,6 +104,17 @@ function build_sketches() # Set the time of the cached core.a file to the future so the GIT header # we regen won't cause the builder to throw it out and rebuild from scratch. touch -d 'now + 1 day' $cache_dir/core/*.a + if [ $mk_clean_core -ne 0 ]; then + # Hack workaround for CI not handling core rebuild for global options + rm $cache_dir/core/*.a + fi + fi + + if [ -s ${sketch}.globals.h ]; then + # Set to cleanup core at the start of the next build. + mk_clean_core=1 + else + mk_clean_core=0 fi # Clear out the last built sketch, map, elf, bin files, but leave the compiled From d7d8a00fa89bab9722a93cc49ef2967e5341ea2e Mon Sep 17 00:00:00 2001 From: M Hightower <27247790+mhightower83@users.noreply.github.com> Date: Fri, 1 Apr 2022 18:33:01 -0700 Subject: [PATCH 28/28] Doc update --- doc/faq/a06-global-build-options.rst | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/doc/faq/a06-global-build-options.rst b/doc/faq/a06-global-build-options.rst index e2996114a4..24a5a12c92 100644 --- a/doc/faq/a06-global-build-options.rst +++ b/doc/faq/a06-global-build-options.rst @@ -89,7 +89,7 @@ Global ``.h`` file: ``LowWatermark.ino.globals.h`` #endif -Aggressively Cache Compiled core +Aggressively cache compiled core ================================ This feature appeared with the release of Arduino IDE 1.8.2. The feature @@ -189,7 +189,7 @@ These two command-line options are the problem: ./arduino --preferences-file other-preferences.txt ./arduino --pref compiler.cache_core=false -Hints for discovering the value of ``compiler.cache_core`` use can be +Hints for discovering the value of ``compiler.cache_core``, can be provided by specifying ``mkbuildoptglobals.extra_flags=...`` in ``platform.local.txt``. @@ -218,6 +218,20 @@ At the time of this writing, when Arduino IDE 2.0 rc5 exits, it leaves the temp space dirty. This keeps the workaround active the next time the IDE is started. If this is an issue, manually delete the temp files. +Custom build environments +========================= + +Some custom build environments may have already addressed this issue by +other means. If you have a custom build environment that does not +require this feature and would like to turn it off, you can add the +following lines to the ``platform.local.txt`` used in your build +environment: + +:: + + recipe.hooks.prebuild.2.pattern= + build.opt.flags= + Other build confusion =====================