Repository navigation
Change case in the root locale everywhere in the main code - #135
Merged
Merged
Conversation
A check, run on every pull request, fails on any toLowerCase() or toUpperCase() with no argument in a Java file under src/main, and two tests set a Turkish default locale, where a capital I lowers to a dotless i. The JSON builder must still write ipdata and ismobile, and the Device Detection cloud engine must still read DeviceId from the cloud's deviceid.
Every toLowerCase() and toUpperCase() under src/main now passes Locale.ROOT, 76 calls in 21 files. Under a Turkish or Azeri default locale a capital I lowered with the default locale becomes a dotless i, so the JSON builder wrote keys the client script never reads, the cloud engines looked for keys the response does not have, and names in the Pipeline builder and the usage sharing filter stopped matching. EvidenceKeyFilterShareUsage gets the same four changes as the usage sharing options change, line for line.
CsvNumbersTest expects the distance CompareOnPrem writes into its CSV and the average MetricsOnPrem writes into its CSV with a full stop under a German default locale. RandomIpTest expects the client IP addresses ShareUsageExample makes up to use the digits 0 to 9 under a Saudi Arabian default locale.
CompareOnPrem and MetricsOnPrem wrote a distance and an average into their CSV rows with String.format and the default locale, so a locale with a decimal comma split one column into two. ShareUsageExample made up client IP addresses the same way, so a locale whose digits are not 0 to 9 gave Evidence no engine can read. Each now formats in Locale.ROOT, and the two CSV numbers are package-private methods so a test can reach them.
Two places needed resolving, both by keeping rc's code and naming the root locale in it, as this branch does everywhere else in the main code. - IPIntelligenceCloudEngine.java conflicted. rc moved the processing into processCloudEngine, which the shared Cloud Aspect Engine logic calls, and skips a property the response leaves out. rc's version is kept, with Locale.ROOT in the weighted property name check, as this branch had it. - The Device Detection examples' KeyHelper merged without a conflict, but rc had replaced the line this branch changed with getNamedKey, whose keyName.toUpperCase() named no locale. The root locale check found it, and it now passes Locale.ROOT. ci/check-root-locale.ps1 passes on the merged tree, with every case change in 533 main source files naming its locale.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Java's
String.toLowerCase()andtoUpperCase()with no argument use the default locale of the JVM. Under a Turkish or Azeri default locale a capital I lowers to a dotless i (ı), and a small i uppers to a dotted capital İ, so a name that holds one no longer matches its lower case form. Onrcat ceab0888, 74 lines undersrc/maincall one of them with no argument (76 calls in 21 files), among them:JsonBuilderElement.javalowers every element and property name it writes, soIsMobilebecameısmobile, a key the client-side script never reads.DeviceDetectionCloudEngine.javaline 381 and its five neighbours, the IP Intelligence and Location cloud engines andCloudAspectEngineBaselower the Property name to find its value in the cloud's lower case response, soDeviceIdlooked fordeviceıdand found no value.PipelineBuilder.javalowers builder and method names from a configuration file to match them, andEvidenceKeyFilterShareUsage.javalowers Evidence keys before it compares them with the keys that are never shared.The specification asks for lower case keys compared without regard to case. json-builder.md lines 59 to 61 at d79bbe3b says "These keys MUST be written in lower case. The key of a top-level entry is the Element Data's Data Key, and the key of a sub-entry is the Property name", and access-to-results.md lines 5 to 11 says "Where string keys are used ... Lower-case SHOULD be used. String comparisons MUST be case insensitive." Lowering in the root locale gives the same key on every JVM, whatever its default locale.
#83 fixed the four calls in
EvidenceKeyFilterShareUsage.java, and this draft does the same for the rest of the main code.What changed and why
src/mainpassesLocale.ROOT, 76 calls in 21 files (d3990945), withimport java.util.Localewhere a file did not import it. Nothing else in those lines changed.EvidenceKeyFilterShareUsage.javagets the same four changes and the same import as Compare usage sharing keys in the root locale #83, line for line. Compare usage sharing keys in the root locale #83 has since merged intorc(b76596b), so that file merges as no change, and what this draft adds torc49bc431 is 73 calls in 20 files, one of them in thegetNamedKeythatrcadded to the Device Detection examples'KeyHelper(see Base branch).ci/check-root-locale.ps1reads every Java file git tracks under asrc/mainfolder and fails on anytoLowerCase()ortoUpperCase()with no argument, naming each line..github/workflows/root-locale.ymlruns its Pester tests and then the check on every pull request and on each push tomainandrc, as the environment variable names check does. Tests undersrc/testare left out, because a test may set a locale on purpose.compare-output.csvwithString.format("%.3f", ...), and MetricsOnPrem.java line 739 ended a comma separated row withString.format("%.2f", ...). Under a default locale with a decimal comma, such as German, 12.346 became 12,346 and the row gained a column.ShareUsageExamplemade up client IP addresses withString.format("%d.%d.%d.%d", ...), so under a locale whose digits are not 0 to 9, such as Arabic in Saudi Arabia, the address was Evidence no engine can read. Each now formats inLocale.ROOT, and the two CSV numbers move into package-private methods,kilometresandtwoDecimalPlaces, so a test can reach them. The examples are not published packages, so this releases nothing.Every published package whose main code changed is released by the version per package change (#63).
Tests
Tests first, in 31153475.
ci/check-root-locale.Tests.ps1has 18 Pester tests. They check that the pattern finds a bare call and leavesLocale.ROOT, another locale,equalsIgnoreCaseandCharacter.toLowerCasealone, that each line number is given, that onlysrc/mainJava files are read, and that the script run on a throwaway repository passes with a named locale and fails, naming the line, without one.JsonBuilderTests.JsonBuilder_TurkishLocale_NamesLowerCasedInRootLocalesets the default locale to tr-TR and builds the JSON for an elementIpDatawith a PropertyIsMobile, and expectsipdataandismobile.MissingPropertyHandlingTests.PropertyWithCapitalI_TurkishLocale_HasValuesets the default locale to tr-TR and has the Device Detection cloud engine readDeviceIdfrom a response that givesdeviceid, and expects the value.Each ran on
ubuntu-lateston a throwaway branch holding the commit and a temporary workflow that ran the Pester tests, the check, andmvn -Pdev -pl pipeline/pipeline.jsonbuilder,device-detection/device-detection.cloud -am testwith only the two Java test classes, on Java 17 as the full build'sUbuntu_Java_17entry does.tmp/java-locale-red). The check named 74 lines and failed with "74 case changes name no locale".JsonBuilder_TurkishLocale_NamesLowerCasedInRootLocalefailed with{"ıpdata": {"ısmobile": "True"}} ==> expected: <true> but was: <false>, andPropertyWithCapitalI_TurkishLocale_HasValuefailed, 1 of the 3 in its class. The 18 Pester tests passed.tmp/java-locale-green). The check printed "Every case change in 525 main source files names its locale", the 18 Pester tests passed,JsonBuilderTestspassed 16 of 16 andMissingPropertyHandlingTests3 of 3.CsvNumbersTest.csvNumbers_GermanDefaultLocale_FullStopsets the default locale to German and expectskilometres(12.3456)as12.346andtwoDecimalPlaceswith a full stop too.RandomIpTest.randomIp_ArabicDefaultLocale_AsciiDigitssets the default locale to ar-SA and expects 20 made-up addresses written with the digits 0 to 9. Both ran onubuntu-lateston Java 17 through a temporary workflow, building each examples folder against the packages on Maven Central. The red, run 43666180249 (tmp/java-exlocale-red2), ran the tests against the fix's methods with the default locale, as the methods do not exist before the fix.CsvNumbersTestfailed with "expected:<12[.]346> but was:<12[,]346>" andRandomIpTestwith the address٢٣٦.١٦٦.٢٢٤.١٥٠. The green, run 43666180345 (tmp/java-exlocale-green2), ran d6263a2 and both passed.Changes at this head
The previous head, bb1d8da, had been built only against
rc32b3784, and byrc49bc431,rchad changed 16 of the 30 files this draft changes, so the draft no longer merged. One commit follows it.rc49bc431 in, as Base branch says. It adds no test, so it has no red. Its green is the tests of what the draft touches, below, and the locale check on the merged code.Tests of what it touches
JsonBuilderTestspassed 18 of 18, the Device Detection cloudMissingPropertyHandlingTests3 of 3,EvidenceKeyFilterShareUsageTests15 of 15,PipelineBuilderTests26 of 26, andCsvNumbersTestandRandomIpTest1 of 1 each. macOS was left out of this build. Therchead is built with macOS before it merges intomain.An earlier full build, run 43666245300, built bb1d8da with
rc32b3784 on the same nine configurations, and every job passed.Released tests
The released tests on
mainthat exercise what this draft touches includeJsonBuilderTests(8 onmain), the Device Detection cloudMissingPropertyHandlingTests(2 onmain),EvidenceKeyFilterShareUsageTestsandPipelineBuilderTests. The draft changes none of them. It adds one test toJsonBuilderTestsand one toMissingPropertyHandlingTests, and the run above passed every test in each class.Base branch
rc49bc431 is merged into the head with a merge commit, 1442b1d3. Two places needed resolving, both by keepingrc's code and naming the root locale in it, as this draft does everywhere else in the main code.IPIntelligenceCloudEngine.javaconflicted.rcmoved the processing intoprocessCloudEngine, which the shared Cloud Aspect Engine logic calls, and skips a Property the response leaves out.rc's version is kept, withLocale.ROOTin the weighted Property name check, as this draft had it.KeyHelpermerged without a conflict, butrchad replaced the line this draft changed withgetNamedKey, whosekeyName.toUpperCase()named no locale. The locale check found it, and it now passesLocale.ROOT.rchas since moved to 3bb6e71 with #158, #173 and #179. None of them adds a case change with no locale undersrc/main, and the draft merges with 3bb6e71 with no conflict.Before you upgrade
Under a default locale such as Turkish or Azeri, the JSON the JSON Builder writes now has
ismobilewhere it hadısmobile, and the cloud engines now find a value for a Property whose name holds a capital I, such asDeviceId, where they found none. The JSON Builder's keys follow "These keys MUST be written in lower case" in json-builder.md (above). Code that read one of the dotless keys under such a locale reads the lower case key instead. Under any other default locale nothing changes.What remains
src/main, so any draft that adds one fails that check once this merges.rcb33daa0f. It needs no change here, and the one case whose output a machine reads is fixed in #139.String.formatcalls name no locale. 21 of them build log, warning and exception messages for a person to read. The 22nd,"%04x"in ShareUsageElement.java line 648, writes an escape into the usage sharing record, and the Formatter documentation says ofxthat "No localization is applied".SimpleDateFormatmade with no locale (ShareUsageBase.java line 66). #139 writes DateSent with aDateTimeFormatterinLocale.ROOTand in UTC, so it is left to that draft.HttpURLConnection.setIfModifiedSince, which takes a number and leaves the date to the JDK (DataUpdateServiceDefault.java line 833).NumberFormat,DecimalFormat,MessageFormatorDateTimeFormatterin the package code is made without a locale.