newRPL - build 1255 released! [updated to 1299]

[Claudio L.]

(12-19-2018 08:04 PM)The Shadow Wrote:  Also... Do we actually *need* to be able to handle matrix equations? OldRPL can't.

Good point, perhaps we don't. However, if we are designing a new CAS from scratch, I'd like to make it useful not just for a kid to get a result that looks like a book, but for more advanced users to make it their own as well.
You mentioned you liked to use the CAS to simplify boolean operations, that's an actual real application, and I'd like this CAS to be able to do it without anything special. Just define a few rules and you should be good to go.
Back in the days of the hp48 I did a lot of work with matrix expressions, and I could've used a CAS to simplify some proofs.
The main thing is that I don't want the automatic simplification to mess up the equations (like putting a 1 where it should've been I, or doing permutations on multiplicands that could be matrices), to at least give a chance to those future user-defined rules to work.

You threw an idea of using names like X.MAT. That's basically what the Prime does, variable names that imply the content based on the letter. What I don't like is that forces one to use an unnatural selection of names on equations. That maybe OK for doing homework, but in physics you want to call variables by their "usual" name (P*V=n*R*T wouldn't mean the same if you change the letters, would it?)

I think we need a way to display the attributes of variables, though. I'd like a variable to be called the same, but look different based on its attributes. For example, bold text might be constant variables, perhaps complex ones could be double-struck, and matrices and vectors could have a line over them (like complex infinity does). We'd need a lot of work on fonts though.

[Claudio L.]

(12-19-2018 10:22 PM)Claudio L. Wrote:  I think we need a way to display the attributes of variables, though. I'd like a variable to be called the same, but look different based on its attributes. For example, bold text might be constant variables, perhaps complex ones could be double-struck, and matrices and vectors could have a line over them (like complex infinity does). We'd need a lot of work on fonts though.

Thinking out loud...
What we have implemented is a 32-bit word of 'attributes' that goes optionally attached to every IDENT object. So we can easily add CAS information about every variable.
Question is how does the user add/edit/delete those attributes?
Of course, in the distant future the equation editor will allow the user to check a few boxes to change the attributes of a variable.
But in reality it should also be:
* persistent across edits in the command line
* persistent when sharing the expression as decompiled source code
* visible on the screen

What if we use the subscript numbers? We have readily available in the keyboard the numbers from 0-9 so it would be easy to type/edit (not easy to remember what the attributes mean, though).
The compiler would recognize these subscripts, remove them from the name and store them in the attribute word. In the same way, the decompiler can extract the bits, convert to decimal and add the text after the variable name.
A code enhancer/pretty print could detect these numbers in the same way and replace them with bold font, etc depending on variable type.
A flag could control if the numbers are actually displayed or not on the stack.

One possible encoding (using 2 digits) could be:
0 = (no subscript) = unknown type, assumed complex by default
1 = real variable
2 = constant (real or complex)
3 = matrix
Second digit (optional additional knowledge about the variable, these are bits):
When first digit is 0,1,2:
1 = value cannot be negative (>=0)
2 = value cannot be positive (<=0)
4 = value cannot be zero
5,6,7 = combinations of previous bits
When first digit is 3 (matrix):
1 = matrix is square
2 = matrix is symmetric (A = AT)
3 = matrix is invertible (INV(A) exists)
4 = matrix is orthogonal (AT = INV(A))

So for example, 'A₃₃*X₃' means A is a matrix (first 3) and is invertible (second 3), while X is a matrix but no other information is known.
For display, a flag could hide the numbers and show just 'A*X', only showing the numbers when editing.
Perhaps a menu could help tick boxes to change the subscripts while in the editor.

[The Shadow]

Wow. Nobody can say you aren't ambitious!

Okay, fine. I have no idea how to do the display stuff, though overline sounds good for vectors and matrices.

Sounds like we need commands RVAR and MVAR that set a variable to be real or matrix respectively? (Hopefully they work on lists.) Do we need a separate vector type? Do we need to distinguish between row and column vectors?

Claudio L. Wrote:Second digit (optional additional knowledge about the variable, these are bits):
When first digit is 0,1,2:
1 = value cannot be negative (>=0)
2 = value cannot be positive (<=0)
4 = value cannot be zero

While this makes good sense for reals, the first two make no sense for complex numbers. For those, it makes more sense to specify the quadrant, but I suppose you could just do the sign of the real part, which would be better than nothing.

I sort-of used the CAS to simplify Boolean expressions. What I actually did was use EVAL to simplify, then use A->L to convert the algebraic to a list, then strip out all exponents, then convert back to an algebraic. MUCH easier to do with RULEAPPLY, of course!

What exactly does RULEMATCH do?

[The Shadow]

Your system in principle would allow for equations involving both real (or complex) and matrix variables. The problem is this will cause the number of rules to balloon.

Maybe your original idea of having a "matrix mode" set by flags is the way to go... In that mode, scalar variables are treated as matrix variables - in essence, just multiplying them by the identity matrix. Rather than a special symbol for the zero matrix for example, you just treat it as the scalar 0 times I. Does that make sense?

As for the size of the identity matrix... You could treat it as a global variable, the way modular bases are now?

EDIT: Your matrix variables could include further settings: Diagonal, and "multiples of the identity matrix". Possibly even triangular.

[The Shadow]

Musing some more... Or perhaps rather than a mode, if any matrix variables are present in an algebraic, then all scalar variables are treated as multiples of the identity matrix.

[The Shadow]

What sort of rule do I need to convince '2*X*Y-Y*X' to become 'X*Y'?

I thought this would do it (when followed with AUTOSIMPLIFY):

'.XN*.xX+.XM*.xX:→(.XN+.XM)*.xX'

since it's commutative. It doesn't work, however.

However, that rule *does* turn '2*X*Y+Y*X' to '(X+2*X)*Y'. Why won't it work on the subtraction?

[Claudio L.]

(12-20-2018 10:17 PM)The Shadow Wrote:  What sort of rule do I need to convince '2*X*Y-Y*X' to become 'X*Y'?

I thought this would do it (when followed with AUTOSIMPLIFY):

'.XN*.xX+.XM*.xX:→(.XN+.XM)*.xX'

since it's commutative. It doesn't work, however.

However, that rule *does* turn '2*X*Y+Y*X' to '(X+2*X)*Y'. Why won't it work on the subtraction?

New rules engine (3rd rewrite) is up and running on my system, I'll test some more before releasing in the wild and also add the variable attributes support we discussed above. Your example rule already produces the correct result '(2*Y-Y)*X' which AUOSIMPLIFY quickly turns into Y*X. The new engine uses much more RAM but does proper backtracking. The previous generation tried to cut corners to save RAM but cases like the one above were failing to test all possible permutations.
AUTOSIMPLIFY already has 8 independent groups of rules, controlled by 8 independent flags to disable/enable the simplifications.
We still have to come up with what the groups mean but the first one is custom user rules (in a variable 'AUTOSIMPLIFY.RULES'), the second group is basic cleanup rules, 3rd group is division/negative powers simplification (this can remove poles from an expression), and the rest are yet to be defined (blank for now, we'll fill them up with community input over the years).

[The Shadow]

What format does AUTOSIMPLIFY.RULES use? Is it just a list?

Thanks for all your hard work!

[Claudio L.]

(01-04-2019 10:04 PM)The Shadow Wrote:  What format does AUTOSIMPLIFY.RULES use? Is it just a list?

Thanks for all your hard work!

Same as for RULEAPPLY and all other commands that use rules, it can contain:

* A single rule
* A list of rules. As a special case, if an ident is included in the list instead of a rule, it's considered to be a variable name to include a set of rules, so you can separate rules in groups (lists), store them in variables, and then have a "master" list that includes those rule groups by name. Notice that including the same rule group multiple times is valid, and will actually perform each rule in the order provided. Also be careful not to create a circular reference of rules.

Edit: By the way, if you include an ident in the list that doesn't exist as a variable, it will be ignored silently.

[The Shadow]

Very nice. Are user rules done before or after the built-in rules? Oh - and have the list of AUTOSIMPLIFY rules changed from the last time you posted them? (I know they've been organized into groups. Maybe document them on the wiki?)

And when will it be ready to be released into the wild?

(It might be handy later, btw, to be able to distinguish between even and odd integers in rules... Maybe .oVAR and .eVAR? Or, if rules can include IFTE expressions, that would do it and then some.)

[Claudio L.]

(01-10-2019 12:43 AM)The Shadow Wrote:  Very nice. Are user rules done before or after the built-in rules? Oh - and have the list of AUTOSIMPLIFY rules changed from the last time you posted them? (I know they've been organized into groups. Maybe document them on the wiki?)

And when will it be ready to be released into the wild?

(It might be handy later, btw, to be able to distinguish between even and odd integers in rules... Maybe .oVAR and .eVAR? Or, if rules can include IFTE expressions, that would do it and then some.)

Once it works and I stop changing things it will be fully documented in the wiki (this will be the foundation for reimplementing many of the 50g commands, so it needs to be well documented).

Algorithm works well now, still squashing the last few bugs (some crashes) and testing corner cases to make sure every subtlety is handled correctly. I'll test it internally at least for another week before letting it out. Failing to process corner cases would be OK for preview, but crashing the calc is not acceptable in my book.

Regarding IFTE: Mathematica calls them "constraints" and aren't exactly IFTE expressions. My thought was to have a single condition expression (then you can use AND() and OR() to pack multiple ones in one), in which you can use the special variables defined. This would be evaluated only when a match is found and is ready to make a replacement (hence all special variables have a definition).
The problem is this expression cannot be evaluated with ->NUM, variables that exist need to be replaced, then any equality needs to be tested by structure, not by value. In other words, we'd have to symbolically simplify this expression until we can check the equality to be true or false (we need to fire up the rules engine from within the engine to simplify the expression within the rule!).
This recursive embedded behavior can be a pain to handle, not to mention make everything slow to a crawl if every time a candidate replacement is found, we need to replace variables, simplify and test the condition.

But, dreaming out loud, what could be a good syntax to include such conditional expression?
Rules are 'Left:->Right'
Perhaps the | (given that) operator is a good choice?
'Left:->Right|Condition'
for example, a condition to test for odd :
'√(.xX^.iN):->.xX^((.iN-1)/2)*√.xX | FP(.iN/2)≠0'

Mmmmm... this would need to be evaluated numerically as well, not just structure... need to think some more about this...

[The Shadow]

Maybe I'm misunderstanding something, but in the case of .iN you've got an actual number. Why can't you use ->NUM on it?

[Claudio L.]

(01-10-2019 03:34 PM)The Shadow Wrote:  Maybe I'm misunderstanding something, but in the case of .iN you've got an actual number. Why can't you use ->NUM on it?

That's just a special case, in the general case the condition could be anything, so equality needs to be simplified and checked symbolically. In purely numeric expressions ->NUM could work, but will throw an error if there's any undefined variables, and even if they are defined, how do we know the user wants the variable replaced? The rules engine shouldn't depend on the values of variables other than the special variables defined during the rule processing. All others should remain symbolic, and there's no way to tell if ->NUM's result pulled the value of a variable or not.

[The Shadow]

I'm trying to come up with situations in which one would define conditions on things other than numeric values, and so far I'm coming up blank.

I suppose there might be cases in which one would condition on the exact form of an expression, but surely that's what the actual rules are for?

Likewise, there might be cases where the exact variable being matched would matter, but every case I can think of would be better handled either by just using that variable in the rule, or by the variable attributes you've discussed. Conditioning on those attributes would indeed be very useful, I trust that is possible? (To, for example, define a rule as only applying to matrix variables.) But they'd still be numeric values.

Is there some category of conditions I'm just not seeing?

[Claudio L.]

(01-11-2019 03:05 AM)The Shadow Wrote:  I'm trying to come up with situations in which one would define conditions on things other than numeric values, and so far I'm coming up blank.

I suppose there might be cases in which one would condition on the exact form of an expression, but surely that's what the actual rules are for?

Likewise, there might be cases where the exact variable being matched would matter, but every case I can think of would be better handled either by just using that variable in the rule, or by the variable attributes you've discussed. Conditioning on those attributes would indeed be very useful, I trust that is possible? (To, for example, define a rule as only applying to matrix variables.) But they'd still be numeric values.

Is there some category of conditions I'm just not seeing?

I can't come up with an example either, but also remember, many numeric expressions can't be computed exactly with ->NUM either, imagine a rule to replace exact known values for SIN(), we need that argument to be expressed as a fraction of pi, which needs to remain symbolic to be exact.
You could write SIN(.NN*pi/2) in the rule itself, but that wouldn't match SIN(pi-pi/2) at first sight. Of course, you could use RULEAPPLY, which would fail at first, then do autosimplification (turning into SIN(pi/2)), then will try again and would match, but not with RULEAPPLY1 which would fail initially and that's the end of it.
Moving that to a condition by writing SIN(.NN) :-> ... | FP(2*.NN/pi)==0 you need to first simplify the condition symbolically so if .NN has pi as a factor it would cancel out, then numerically to check that the result is an integer without precision loss.

[The Shadow]

Ah! I see your point. Yes, that would be a very realistic sort of condition to use.

[The Shadow]

I've been thinking some more about equations involving matrices.

Many headaches arise when the matrices aren't square. There's whole classes of things you can't do to them that you can to square ones: inversion and raising to a power, among others. (Though a "pseudo-inverse" can be defined for all matrices, whether rectangular or non-singular, and might be worth implementing.)

But worse yet, there's *three* versions of the identity matrix you'd need for interacting with scalars: one for multiplying from the left, one for multiplying from the right, and one used for addition.

Does anyone actually use non-square matrices in equations? The only examples that are immediately coming to mind are ones in which they're multiplied by their transpose to make them square first.

Any new news, Claudio?

[John Keith]

(01-20-2019 07:03 PM)The Shadow Wrote:  Does anyone actually use non-square matrices in equations?

The outer product / Kronecker product comes to mind.

[pier4r]

Quote:So I think it's easy to add a second type of operation engine

If you think it is easy, since newRPL is open source, simply produce a pull request helping Claudio.
Everything is easy when it costs only one sentence to the one requesting it.

[Claudio L.]

(01-20-2019 07:03 PM)The Shadow Wrote:  I've been thinking some more about equations involving matrices.

Many headaches arise when the matrices aren't square. There's whole classes of things you can't do to them that you can to square ones: inversion and raising to a power, among others. (Though a "pseudo-inverse" can be defined for all matrices, whether rectangular or non-singular, and might be worth implementing.)

But worse yet, there's *three* versions of the identity matrix you'd need for interacting with scalars: one for multiplying from the left, one for multiplying from the right, and one used for addition.

Does anyone actually use non-square matrices in equations? The only examples that are immediately coming to mind are ones in which they're multiplied by their transpose to make them square first.

Any new news, Claudio?

Regarding matrices, I decided to cut the details: we'll identify a variable as a matrix, that's all. This would allow the CAS to not mess the equation by applying commutative rules, etc. but that's all it would do. I don't think the system should apply automatically any rules, we can have special commands for that (like TRIGSIN and friends, commands that all they do is apply a specific rule to an expression).
In the future we can revisit this to add more attributes if we think it's justified.

As far as news: not much, still testing the new rewrite of the engine (works great so far), and adding the needed support for variable attributes that we discussed. The main challenge is to "compute" attributes for expressions, which is quite involved and I'm working on it right now.
For example, let's have an expression like X^(2+2). You want a rule that matches an expression known to be positive (>0). You can specify this in the rule by using attributes in the variable name '.xX₂₃' (here ₂₃ represents a real variable that cannot be infinite and is known to be >0). Now we need the rules engine to analyze the expression X^(2+2) to compute some attributes to see if we can match.
It starts by setting attributes for the constants, 2 is integer, even, >0, non-infinite. Added to the other 2 gives you another integer, even, >0, non-infinite. Then we get the attributes from X (if none given it's assumed finite and real, or finite and complex if complex mode enabled). Now we have a finite real raised to an integer, even power, therefore the engine needs to be smart enough to know this will result in a real, finite, >=0 result.
In this case it won't match, since we want an expression >0, and >=0 doesn't qualify, but my point was to show the kind of inference that this engine needs to be capable of in order for this to be useful.
I'm working case by case all different combinations of all operators to see what conclusions we can get from an expression (and it's taking me a while).