Commit graph

2929 commits

Author SHA1 Message Date
HanishKVC
989c6c4125 SimpCfg: Cleanup the Note a bit to avoid some ambiguities 2024-05-06 11:27:56 +05:30
HanishKVC
344c068d7b SimpCfg:MultiPart keys wrt get_vector
With this and past few commits, now there is simple yet sufficient
support to help move multi-level-hierarchy config files into the
SimpCfg's simple physically 1-level, but if reqd logically multi
level hierarchy flow.

B4 this series of commits also one could have still achieved this,
but there would have been bit more effort needed.
2024-05-06 11:27:56 +05:30
HanishKVC
19d3c88e8a SimpCfg:MultiPart keys wrt get_value etal 2024-05-06 11:27:56 +05:30
HanishKVC
623d0b60da SimpCfg: General MultiPart support, KeyParts not Key wrt SetValue 2024-05-06 11:27:56 +05:30
HanishKVC
c6ecd9316e SimpCfg: Use to_str instead of using stringstream directly 2024-05-06 11:27:56 +05:30
HanishKVC
5380b1e86e ChatON:Update meta.json wrt command-r models
template info picked from tokenizer config's default entry,

Verified that same is used in the existing hardcoded chat apply
template flow.
2024-05-06 11:27:56 +05:30
HanishKVC
2b14bcaddb SimpCfg:ChatON: add by Humans for All note 2024-05-06 11:27:56 +05:30
HanishKVC
20e5b383c5 SimpCfg:Trim DumpHexString only if SC_DEBUG_VERBOSE 2024-05-06 11:27:56 +05:30
HanishKVC
f53c19baac SimpCfg: Update the notes wrt tolower and add test code 2024-05-06 11:27:56 +05:30
HanishKVC
3287fdba28 SimpCfg:Fix/cleanup trim related test samples and flow
Use the commonality between Indian languages to show mixup issue
with the simple minded trim_dump logic and how trim_oversmart
could potentially avoid that.

Given that I am using valid strings to show the pitfalls of fixed
native char size driven logic, so no need to keep the dump and
oversmart flows seperate, so merge into a common loop.
2024-05-06 11:27:56 +05:30
HanishKVC
33619a3b92 SimpCfg: Templatize str_lower 2024-05-06 11:27:56 +05:30
HanishKVC
32ba195a83 SimpCfg: Templatize str_trim_single
Also use NativeCharSize and MultiNativeCharSize wording to make
the note more generic
2024-05-06 11:27:56 +05:30
HanishKVC
5b8bf849c0 SimpCfg: Fixed & ~Variable Length to Native & MultiNativeCharSize
So as to make the notes, more generic.
2024-05-06 11:27:56 +05:30
HanishKVC
d030a26f3c SimpCfg:Update TrimOverSmart use templated TrimDumb after wstrconv 2024-05-06 11:27:56 +05:30
HanishKVC
97ac443bba SimpCfg:Cleanup, updated notes, templated code
Update the notes to match the templated flow now and some of the
nitty gritties involved.

Update DumpHexString to be templated.

Split check nonenglish flow wrt trim dumb and oversmart testing,
so that things with work with one, but not the other can be
differentiated in the flow.
2024-05-06 11:27:56 +05:30
HanishKVC
bf111a83f1 SimpCfg:TemplatedDumbTrim; Test dumb and oversmart trim logics 2024-05-06 11:27:56 +05:30
HanishKVC
554b00f027 SimpCfg: Add some missing const refs 2024-05-06 11:27:56 +05:30
HanishKVC
cae0fff715 SimpCfg: Update notes; Try add a better trimming logic 2024-05-06 11:27:56 +05:30
HanishKVC
d1156cc055 SimpCfg: As locale manipulation reqd for better processing 2024-05-06 11:27:56 +05:30
HanishKVC
2325764180 SimpCfg:CheckStrings: Switch Mbs2Wcs to multithread safe calls 2024-05-06 11:27:56 +05:30
HanishKVC
23acf07bb2 SimpCfg:CheckStrings: Cleanup wstring flow to needed parts 2024-05-06 11:27:56 +05:30
HanishKVC
2cda78f1ad SimpCfg:CheckStrings: WString2String finally
The constructor method doesnt convert wstring to string, when it
involves non-english chars which will encode to multibyte chars
in utf8. even thou it does work for the already utf8 u8string.

wcstombs doesnt seem to work for non english chars, when the
locale is set to the default c, need to change to something like
en_US.UTF-8, to allow it to do the conversion properly.
2024-05-06 11:27:56 +05:30
HanishKVC
7607dbc8c7 SimpCfg:CheckStrings: Try fixup wstring handling 2024-05-06 11:27:56 +05:30
HanishKVC
1a618a42f8 SimpCfg: Update the func notes with alert 2024-05-06 11:27:56 +05:30
HanishKVC
66d6fa62b7 SimpCfg: C++ and strings is a mess even after decades
Seperate out the checks wrt different string types.

Add a wstring_basic, which verifies that wstring iterator handles
non english chars propery or atleast better.
2024-05-06 11:27:56 +05:30
HanishKVC
3ad5cec47e SimpCfg:CheckStrings:MacOS, wstring and wcout
Without using imbue, I couldnt get non-english wstrings to print
on mac. Need to check on linux also.

Also avoid the uint8_t typecasting, given that wchar isnt 8bit
2024-05-06 11:27:56 +05:30
HanishKVC
a448fec486 SimpCfg:CheckString: organise and probe - p3 wstring
wcouts' involving 2nd wstring with non english char in it not
showing up?
2024-05-06 11:27:56 +05:30
HanishKVC
691d0d43b5 SimpCfg:CheckStrings: Organise and Probe - P2 - std::u8string 2024-05-06 11:27:56 +05:30
HanishKVC
713520caac SimpCfg:CheckStrings: Organise and Probe - p1 std::string 2024-05-06 11:27:56 +05:30
HanishKVC
56f19c7a68 SimpCfg: Test c++ string handling 2024-05-06 11:27:56 +05:30
HanishKVC
86e776c857 SimpCfg: Rename to get_vector, add some test code 2024-05-06 11:27:56 +05:30
HanishKVC
1e1f54ec1d SimpCfg:GetArray flesh out, helpers to convert to string
Good: Forgotten love, All the light we cant see, laapataa ladies
2024-05-06 11:27:56 +05:30
HanishKVC
561f50930e SimpCfg: initial go at adding support for spreadout arrays
By having a seperate entry for each element of the array, the
existing logic itself can be repurposed with minimal additions.
2024-05-06 11:27:56 +05:30
HanishKVC
8fdc80533f SimpCfg:Cleanup:Cmdline Arg, GetValueCallerLogging, StringCmp
Bring the direct string comparison also back, as the issue with
cmp was more to do with string.transform.
2024-05-06 11:27:56 +05:30
HanishKVC
08b9711d68 SimpCfg:Remove dbug logs wrt str_tolower and set_bool
Also the warning wrt is it string, now also logs the line number,
group and key, to help user identify the line better.

Misc: pass time last week Another life, Anchakkallakokkan, Deadloch
2024-05-06 11:27:56 +05:30
HanishKVC
0e0d7da18f SimpCfg:Found issue with str_tolower, transform doesnt resize dst 2024-05-06 11:27:56 +05:30
HanishKVC
eb56517f20 SimpCfg:Bools:Make lowercase b4 checking true/false for bool path
TODO: string check wrt true/false, doesnt seem to be working after
str_tolower was introduced. I seem to be doing some silly mistake
not able to make out, moving in and out of sleep, need to check
tomorrow.

string == string-literal failed
string == string-view failed
string.compare(string-literal) failed

Bit strange
2024-05-06 11:27:56 +05:30
HanishKVC
ef5a2cf391 SimpCfg:Dbug why bool is not setting properly 2024-05-06 11:27:56 +05:30
HanishKVC
5aa1072aac SimpCfg: Move dump into its own func, Avoid KV iter wrt Get 2024-05-06 11:27:56 +05:30
HanishKVC
6b475e444f SimpCfg: Log Caller of Set/GetValue 2024-05-06 11:27:56 +05:30
HanishKVC
f05f71bdc4 SimpCfg:SetBool string value, str_tolower, SetValueTypeLogging 2024-05-06 11:27:56 +05:30
HanishKVC
1dc7fd0e85 SimpCfg:WIP:Variant TypeDef, to_str and std::get
Cleanup the use of the variant

Initialize and << op stringstream seperately.
2024-05-06 11:27:56 +05:30
HanishKVC
ee1a62c876 SimpCfg:WIP:Switch to C++ Variant type - initial skeleton 2024-05-06 11:27:56 +05:30
HanishKVC
7302b3ab36 SimpCfg: Use stderr wrt internal Log messaging helpers 2024-05-06 11:27:56 +05:30
HanishKVC
a09571318a ChatON: meta-dump returns flag inturn returned by meta-ok
test-chat-template-chaton now tries to check if meta-ok is ok wrt
the template-id being looked into.

Log template-id info also, where it was previously missed out.
2024-05-06 11:27:56 +05:30
HanishKVC
a108000cfd ChatON:Phi3: Add template details to detailed meta.json 2024-05-06 11:27:56 +05:30
HanishKVC
44c05305d0 SimpCfg: Add support for get_double 2024-05-06 11:27:56 +05:30
HanishKVC
8ad2c17e5d SimpCfg: get_int64 logic 2024-05-06 11:27:56 +05:30
HanishKVC
000245b8e8 SimpCfg:Warn possible nonstring strings, some invalid floats
Warn if something not starting with double quote is being treated
as a string.

Show some examples of invalid floating point values wrt this
logics floating point determination code
2024-05-06 11:27:56 +05:30
HanishKVC
a6648b02f2 SimpCfg:Show floating point values in normal and exponential form 2024-05-06 11:27:56 +05:30