When I'm feeling frustrated about my child's lack of Torah and lack of engagement, I have a few approaches.
1. Learn more myself, study the area myself, work on my own learning/spirituality/middos instead of imposing that onto my child
The concept "כל הפוסל במומו פוסל" is at work here. If I'm accusing my child of not davening, then work on my own tefila. If I am concerned my child isn't engaged enough in learning Torah, then engage myself in Torah learning. (This might be a good idea for me, who is challishing to learn Shmuel II with Chana). Leave my child alone and address any inner insecurities and deficiencies I might be projecting onto them by dealing directly with the source.
This also has the added benefit of having Torah more "shagur b'fiv" (fluent on my lips) and it makes Torah come up more naturally in conversation in a way that children can easily relate to. If I have just read about Shimshon, for example, odds are that when my child asks if a human being can knock down those columns of the house we are next to, I will know the story well enough to tell it, instead of having him sit down or be bored or on to the next thing before I look it up.
Let's say my desire for them to learn doesn't come out of my own issues or insecurities, but from a genuine desire to pass knowledge down to them.
Pushing them or pressuring them is probably going to be counterproductive anyway, so even if my heart is purely motivated, practically a lot of what I do is going to be perceived as annoying or pressuring. So I can pour that energy, that fear, that anxiety, that passion out in tefila. The evaluation aspect of tefila really helps me focus on what my goals and priorities are. What my fears are. And where I want to put my energy. And the emotional aspect of tefila is very relieving.
In my recent tefilos, I realized that I desire very strongly that some day Chana will want to learn. So that is where I put my focus. Every interaction with her is in the context of my desire that she will one day want to learn. This brings some clarity and makes it a lot easier to refrain from activities or comments that would be counter productive to that goal. I'm not sure if there is anything I can do to achieve that goal, but it definitely clarifies for me things I shouldn't be saying or doing and "First do no harm" is one of the my habitual motivating mottos.
In my recent tefilos, I also discovered that I'm delighted with Elazar and his progress and learning at the moment, but I fear greatly that he will not be interested in Torah in the future. So that also affected how I am relating to him now, and helped me relax in the now and also gives me some focus and clarity for the future.
3. Make the relationship the most important thing
I've written before about choosing the relationship over academics. This is a slightly different angle. I came to this idea in the context of when I'm time crunched (hello, Tishrei and Nisan) and how I often used to try to cram in academics while I was short tempered and stressed and how I felt that did more harm than good. But now I'm thinking about it in terms of frustrations about the children not learning or fears that they won't learn. I try to keep in sight that the important thing here is my relationship with my child. I want that gut feeling that they have when they walk into the room and see me to be positive. Not irritation. Not disgust. Not stress. Not annoyance. Just glad to see their mom. And maybe, if I'm lucky, to share with me something that they've been thinking about or doing, because they know I'm interested. This comes so naturally when they are young, yelling, "Mommy, look!" a thousand times a day. And it gradually erodes when we start having conflict about their "responsibilities" and I am asking them to do things they don't want, and I'm nagging...Add in that I'm conveying that they should be different in terms of their learning and how they spend their time and that doesn't make them overjoyed to see my face.
So when I have in my head that the important thing here is a positive relationship, it helps me be more careful about what I say, how I say it, what attitude I have when I engage with my child, and what my overall principles are with respect to managing this emotionally difficult situation.
They are more likely to seek my wisdom if they like me and don't dislike me. We are more likely to have conversations if the conversations are pleasant and not yucky. I find myself chanting to myself numerous times through these phases, "Worry about the relationship, nothing else."